Struct PersistentSubprocessController 

pub struct PersistentSubprocessController {

    tab_id: String,
    block_id: String,
    inner: Arc<Mutex<PersistentInner>>,
    broker: Option<Arc<Broker>>,
    event_bus: Option<Arc<EventBus>>,
    mstore: Option<Arc<Store>>,
    filestore: Option<Arc<FileStore>>,
    id_store: Option<Arc<Store>>,
    identity_store: Option<Arc<Store>>,
    auth_key: String,
    health_monitor: Arc<TurnActivityTracker>,
    stdout_seq: Arc<AtomicU64>,
    self_ref: Mutex<Option<Weak<Self>>>,
    agent_id: Mutex<Option<String>>,
    stable_agent_id: Mutex<Option<String>>,
    stable_agent_uid: Mutex<Option<String>>,
    current_segment: Mutex<Option<String>>,
    deferred_drop_report: DeferredDropReport,
    lease_store: Option<Arc<LeaseStore>>,
    boot_id: Arc<str>,
}

PersistentSubprocessController keeps a long-running CLI process alive, sending user messages as NDJSON lines on stdin.

Fields

tab_id: String

block_id: String

inner: Arc<Mutex<PersistentInner>>

broker: Option<Arc<Broker>>

event_bus: Option<Arc<EventBus>>

mstore: Option<Arc<Store>>

filestore: Option<Arc<FileStore>>

FileStore for write-through persistence of output lines (Phase 1.3).

id_store: Option<Arc<Store>>

Dependencies for start()’s eager-resume path (SPEC_PERSISTENT_CONTROLLER_EAGER_RESUME_ON_RECONNECT_2026_09_20.md). Set via with_identity_stores(), not new() — see that method’s own doc comment for why this is an opt-in builder step rather than a required constructor argument.

identity_store: Option<Arc<Store>>

auth_key: String

Empty when never supplied (every existing new() caller) — start() treats an empty auth_key as “identity stores unavailable” the same as id_store/identity_store being None, since a real spawn always has a real key.

health_monitor: Arc<TurnActivityTracker>

stdout_seq: Arc<AtomicU64>

Monotonic counter bumped for every stdout line (including control frames). The AskUserQuestion dead-air fallback snapshots this before sending the answer and re-checks after a short window; any increment means the CLI produced output (assistant content OR a follow-up control_request), i.e. the turn resumed. Counting all frames — including control frames the reader otherwise skips — avoids a spurious fallback when the resumed turn’s first activity is a tool-permission round-trip.

self_ref: Mutex<Option<Weak<Self>>>

Weak self-reference for the stale---resume-session retry (see retry_after_resume_failure) — set by set_self_ref right after construction. The process-waiter task (a detached tokio::spawn that only captures cloned Arc fields, never &self) needs a way to call back into an instance method once the underlying process actually exits; a Weak avoids a reference cycle (this same struct’s spawn_process is what schedules that task).

agent_id: Mutex<Option<String>>

This controller’s own muxbus/jekt identity, captured once at spawn time from muxbus_agent_id_from_env — the independent source of truth Controller::agent_id() exposes for inject_message_inner’s recipient-identity check. Deliberately its own field, not part of PersistentInner — config (and its env_vars, where the identity is read from) isn’t available until spawn_process, well after new(), and this value doesn’t participate in any of PersistentInner’s carefully-ordered spawn/resume/queue state transitions. None for a persistent block with no muxbus identity set (a non-agent process).

stable_agent_id: Mutex<Option<String>>

The STABLE jekt identity (AGENTMUX_AGENT_ID), captured once in spawn_process and never overwritten afterward — unlike agent_id above, which input.rs’s Register-tail deliberately refreshes every turn to track the live, renameable display name (#2697). Exists specifically so Controller::stable_agent_id() can back the jekt registry’s alias entry and the #2695 recipient-identity check even after agent_id has moved on to a post-rename value. See INCIDENT_2026_09_09_JEKT_STABLE_ID_ALIAS.md.

stable_agent_uid: Mutex<Option<String>>

Identity M2: AGENTMUX_AGENT_UID from the spawn env, captured at spawn exactly like stable_agent_id. None until spawned, or when the spawn env carried no UID (no db_agents row yet). See Controller::stable_agent_uid.

current_segment: Mutex<Option<String>>

The segment the latest spawn recorded (segments.rs), so a resume retry can tell the agent’s previous segment from the one that just failed. None until a spawn with an agent UID.

deferred_drop_report: DeferredDropReport

Reports whatever is still deferred when this controller is dropped. See DeferredDropReport.

lease_store: Option<Arc<LeaseStore>>

Where this agent’s single-live-instance lease lives, and this srv’s boot id (the lease owner). Set via with_agent_lease_store(); None (tests, a registry that can’t be opened) means no lease is taken.

boot_id: Arc<str>

Implementations

impl PersistentSubprocessController

pub(super) fn try_claim_eager_resume_spawn_for( &self, restart_generation: Option<u64>, ) -> bool

[Self::try_claim_eager_resume_spawn], and for a config restart’s replacement (Some(generation)) also require that restart’s token (config_restart_generation) in the SAME acquisition that takes the claim, and record the claim as that replacement (restart_spawn_for). The token is not consumed here: spawn_process consumes it right before starting the child, so a Stop during the credential gate in between still cancels the replacement (codex P1s on #3990).

pub(super) fn try_eager_resume( &self, block_meta: &HashMap<String, Value>, session_id: &str, ) -> EagerResumeOutcome

start()’s eager-resume attempt for a pane with a captured agent:sessionid. Never returns an error — every failure mode here is “fall back to lazy”, not “fail the resync”; see start()’s own doc comment for why.

pub(super) fn try_eager_resume_as( &self, block_meta: &HashMap<String, Value>, session_id: &str, restart_generation: Option<u64>, ) -> EagerResumeOutcome

Self::try_eager_resume, optionally as a config restart’s replacement: see Self::try_claim_eager_resume_spawn_for.

impl PersistentSubprocessController

pub(super) fn settle_eager_spawn_failure( &self, err: &str, retry_config: PersistentSpawnConfig, )

try_eager_resume’s spawn-failure settlement, split out so the decision is unit-testable without a real child process. Also the settlement for a refused candidate respawn in respawn_once_for_leftover_queue (codex P1 on PR #3551, sixth round), which fails the same way for the same reason.

Mirrors release_spawn_claim_and_drain_queue’s !spawn_succeeded branch: if something queued during the attempt it needs a live process to go to, so hand off to the fallback respawn; otherwise just release the claim — there’s nothing to drain. Decided under ONE lock acquisition, same as the success branch and for the same reason (codex P2 on PR #3523): as two, a send arriving in the gap saw the claim still held, queued its prompt and returned success, and the release then never rechecked — stranding an accepted prompt until some unrelated later send happened to become the next spawner.

One failure class is exempt from the fallback (codex P1 on PR #3551): session_held_elsewhere refusing the --resume because the same conversation is open — or still closing — in another pane. A fresh spawn there would deliver the accepted prompt to a blank conversation, bypassing the very guard spawn_process just applied. Instead: release the claim, DISCARD the queue, keep the session id, and tell the user which prompts were not delivered and why.

Discarded, not retained (codex P1 on PR #3551, third round): keeping the prompts queued — the credential-gate decline’s shape — only defers the bypass. The next send would become the spawner, hit the same refusal, and release_spawn_claim_and_drain_queue’s generic failed-spawn path, which knows nothing about WHY, would see the leftovers and hand them to respawn_once_for_leftover_queue — a fresh, blank conversation after all. The only way to make that path unable to misread them is for them not to be there.

impl PersistentSubprocessController

pub(super) fn restart_spawn_still_permitted_locked( inner: &mut PersistentInner, ) -> Result<(), String>

Called under inner just before spawn_process starts the OS child: if this spawn is a config restart’s replacement and a Stop cleared its token since the claim (the claim precedes the credential gate, which can be slow), refuse, so no process is started at all. A cheap early out only: the token stays live, because a Stop can still land while the child is being started, and only Self::commit_restart_spawn_locked, in the acquisition that installs the child, decides (codex P1s on #3990). Refusing returns an error, so the caller takes its ordinary spawn-failure path: the claim is released, and a message a human typed meanwhile still gets its own spawn.

pub(super) fn commit_restart_spawn_locked( inner: &mut PersistentInner, ) -> Result<(), String>

Called in the SAME inner acquisition that installs a freshly started child as the live process: consume a config restart’s token, or refuse if a Stop cleared it while the child was starting. Until this point a Stop has no kill_tx to reach the new process through, so this is where it must win: on refusal the caller kills the child it just started and fails the spawn.

pub(super) fn config_restart_due_locked( inner: &PersistentInner, generation: u64, ) -> bool

Whether the process of generation just ended as a committed config restart that nothing has overridden, so its exit handler must bring the replacement up. Checked by BOTH exit arms: the process can exit on its own right after its last result and win the select! before the kill is consumed (codex P1 on #3990). A read, not a take: the token is consumed only right before the replacement’s child starts, so a Stop until then still wins.

pub(super) fn respawn_after_config_restart(&self, generation: u64)

The second half of a deferred runtime-config restart, run by the old process’s exit handler once it is gone: resume the same session with the config now in block meta, exactly as a restart that found the pane idle does (resync_controller replaces the controller and start() eager-resumes). Before this the pane stayed empty “until the next message”, and a jekt is never that message: automated delivery doesn’t spawn, and it waits while restart_pending is set, so every message to the agent queued until a human typed into it (docs/retro/RETRO_DEFERRED_RESTART_NEVER_RESPAWNS_2026_09_29.md).

Nothing is started for a controller that’s no longer this block’s (a pane closed or replaced in the window), when something already replaced the process (any spawn clears the token), or when a Stop overrode the restart: the token is consumed only inside the spawn claim, under the lock a Stop clears it with. Whenever no replacement comes, the restart window is closed anyway, so queued deliveries fall to the no-process rules (reported stranded after the watchdog’s grace) instead of waiting for a spawn that isn’t coming.

pub(super) fn close_restart_window_without_process(&self)

End a config restart’s quiesce window when no replacement is coming: with restart_pending (and the restart’s own stop_pending) left set, deferred_must_wait_locked holds every queued delivery forever and the watchdog never reports it. Leaves them alone if a spawn has meanwhile begun, since that spawn clears both itself.

impl PersistentSubprocessController

pub(super) fn redelivery_persister(&self) -> RedeliveryPersister

What a dead-air fallback needs to record its re-delivered line — see RedeliveryPersister.

pub(super) fn encode_user_message(message: &str) -> String

Encode a message as the stream-json stdin line the CLI expects.

pub(super) fn try_write_stdin_locked( inner: &mut PersistentInner, line: &str, ) -> Result<(), String>

Write an already-encoded stdin line, given a held inner guard.

Callers must hold the lock across the must-wait check, the queue update and this write, so no concurrent sender or watchdog tick can slip between them.

On success the line is also tracked into the current generation’s resume retry batch, so every write site (the send path and the watchdog) gets it without having to remember to.

pub fn send_user_message_outcome_from( &self, message: String, origin: Option<TurnInput>, ) -> Result<SendOutcome, String>

[Self::send_user_message_outcome], saying who the input is from (turn provenance, SPEC_AGENT_SELF_QUIT §6.3) — the Swarm broadcast, which is the user’s own message (SPEC_SWARM_BROADCAST_AS_USER_MESSAGE §6 q2). The label is applied only when this message is the only one written: a backlog written with it was queued by automated senders, and a message left queued (process starting, restarting or stopping) is written later by the drain with whatever else is queued. Both stay unlabelled, as every automated delivery is, so the label can never vouch for input that isn’t the user’s.

fn mark_turn_active_locked(&self) -> bool

Mark the turn active while the caller already holds inner, so the flip is never behind the write that caused it. Lock order is inner → health_monitor, matching the turn-end handler in stdout_reader.rs; health_monitor never takes inner, so this cannot deadlock. Returns the pre-call value.

fn mark_turn_active_locked_from(&self, origin: Option<TurnInput>) -> bool

Self::mark_turn_active_locked with the input’s provenance; None is exactly the unlabelled mark.

pub(super) fn deferred_must_wait_locked(&self, inner: &PersistentInner) -> bool

Whether a queued message must keep waiting rather than be written now. Only states in which the process cannot take a write: what the agent is doing (mid-turn or idle) is not one of them (SPEC_JEKT_IMMEDIATE_DELIVERY_2026_09_28.md §2.1). Shared by the send path and the watchdog, so the two can never disagree.

• Another writer owns stdin — see Self::stdin_owned_by_another_writer; this includes a spawn in flight, whose write try_write_stdin_locked would refuse.
• A committed config restart (restart_pending), but only while the process being restarted still holds stdin_tx: it is about to be killed, so writes must wait for the replacement. Once it has exited, the ordinary rules apply. The exit handler’s respawn (respawn_after_config_restart) holds the spawn claim, so the queue keeps waiting for it (a spawn in flight, below), and if no spawn is in flight, whoever was to start one declined or failed. Waiting on restart_pending alone then held jekts forever, since only a successful spawn clears it (Korp, 2026-09-29; ReAgent P1 on #3990: a competing spawn claim that fails leaves it set). The watchdog’s grace window covers the moment between the exit and the respawn’s claim.
• A requested kill (stop_pending), but only while the dying process still holds stdin_tx: writes are refused then, and a refusal must not reach an automated caller as an error (reagent P1 on #3562). Once it exits, the ordinary no-process rules apply. A send fails into the caller’s respawn fallback, and the watchdog’s grace window runs, so nothing parks unreported behind a stop.

fn stdin_owned_by_another_writer(inner: &PersistentInner) -> bool

Another writer is feeding stdin, and messages it carries were accepted earlier than anything in the deferred queue, so they must not be overtaken:

• a spawn is in flight — its seed message and drain go first;
• a stale-resume retry batch is being replayed (drain_claim), possibly into this same live process by a background task writing to the same channel;
• human messages are queued behind a spawn.

Queued messages count only while a process exists to drain them. A drain’s “second stall” (drain_queue_with_claim) releases its claim with leftovers queued and no process. Nothing owns that backlog, and treating it as a writer made a send defer instead of failing into the no-process fallback, and kept the watchdog resetting its orphan timer forever (codex P1 on #3562).

Also part of Self::deferred_must_wait_locked, which every caller of Self::flush_deferred_locked checks first under the same lock.

pub(super) fn flush_deferred_locked( inner: &mut PersistentInner, block_id: &str, ) -> (Vec<String>, Option<String>)

Write the queued messages to stdin, oldest first, as far as stdin takes them. Returns the lines written, in order, and the error that stopped it early, if any: the entry that failed stays at the head of the queue with everything behind it, so a failed write never drops a message.

The whole backlog at once, not one at a time: nothing is held for what the agent is doing any more (SPEC_JEKT_IMMEDIATE_DELIVERY_2026_09_28.md §2.1), only for whether the process can take a write.

The caller holds inner across this, has checked Self::deferred_must_wait_locked under that same guard, and marks the turn active before dropping it if anything was written.

pub(super) fn turn_boundary_locked( inner: &mut PersistentInner, health: &TurnActivityTracker, block_id: &str, generation: u64, stats: Option<PassStats>, ) -> Option<bool>

The turn-boundary decision for a result frame, under inner: the turn is over, and a deferred runtime-config restart may now apply. Returns whether it does.

Returns None, touching nothing, when generation is no longer the current one. A fallback respawn can install a new process before the old reader drains a buffered result. That result ends nothing the new process is doing, so acting on it would mark the replacement’s running turn idle, or restart it (codex P1 on #3562).

The deferred restart applies only when nothing is queued. Killing the process with an entry still queued would strand it; the flag stays set, and the next result (the watchdog’s write starts a turn) applies it.

stats: the result frame’s figures. They go to the turn the pass belonged to in the same step that idles it, so input arriving right after (which may open a new turn) can never take them (#4492).

pub(super) fn ensure_deferred_watchdog(&self)

Make sure a watchdog task is looking after a non-empty deferred queue.

Messages are queued only while the process cannot take a write (see Self::deferred_must_wait_locked), and nothing else writes them once it can, so this is how every queued message gets out:

• a message queued behind a spawn — delivered once the process is up, whether or not that spawn starts a turn (eager resume spawns with no seed message);
• a message queued behind a committed restart or a requested stop — delivered to the replacement process;
• a write that failed (e.g. the bounded stdin channel was momentarily full) — retried;
• a message queued behind a spawn that then FAILED, or behind a process that exited and was never replaced — reported stranded after the grace window.

Idempotent: at most one watchdog per controller. A no-op for an instance without set_self_ref (only tests construct those) or outside a Tokio runtime.

pub(super) fn sweep_deferred_once( &self, orphaned_ticks: &mut u32, ) -> WatchdogStep

One watchdog tick. Split out of the task so tests can drive it directly, without a real process or real time.

pub(super) fn log_stranded_deferred( block_id: &str, reason: &str, stranded: &[String], )

Report every stranded deferred entry, so a message this controller accepted is never quietly discarded (SPEC_NO_MIDTURN_DELIVERY_2026_09_23.md §4.5). Callers drain the queue in the same inner acquisition that ends delivery (stop, shutdown, the watchdog), so no flush can take an entry first.

Known gap: reporting is currently a loud structured log, not a failure routed back to the original sender. The queue stores encoded stdin lines with no sender identity attached, so there is nothing to address a reply to; carrying that identity through is Phase 3. Until then a stranded message is visible in the logs rather than in the sender’s response — better than silence, short of the spec’s intent.

Takes block_id rather than &self so shutdown’s 'static future, which holds no controller reference, can report through it too.

pub(super) fn append_delivered_message(&self, json_str: &str)

Persist a delivered injection to the blockfile WITH a live event.

Unlike send_message there is no agent-message-accepted pending echo to pair with (nothing was typed in the UI), so without this the injection is invisible to the human operator: a silent injection, which SPEC_JEKT_SECURITY_AND_VISIBILITY §3.1/G1 forbids. The live append renders it in the open pane; the persisted line lets parseHistoryLines rebuild the node on reopen.

Called at delivery time, not enqueue time, so the transcript shows the message where the agent actually received it. A queued message is therefore not yet visible — surfacing “N waiting” is a UI question tracked as spec §8 Q3.

pub fn answer_question( &self, tool_use_id: String, answers: Value, ) -> Result<(), String>

Answer a parked AskUserQuestion via the Agent SDK control protocol.

The CLI asked us with a can_use_tool control_request (parked in pending_questions by the stdout reader); we reply with a control_response carrying updatedInput.answers. This is the ONLY mechanism the CLI accepts — delivering a tool_result on stdin does NOT work (the CLI auto-rejects AskUserQuestion within the turn). answers is the JSON object mapping each question’s text to the selected label(s) or free-text. Process must already be running (agent is mid-turn, blocked on this answer). Spec: docs/specs/SPEC_AGENT_CONTROL_PROTOCOL_2026_06_15.md §2.3. pending_questions is in-memory-only, scoped to THIS controller instance — a fresh instance (pane reopen, or any process respawn) starts with an empty map even though the persisted transcript can still show the question as the tail node (deliberately preserved by scrubOrphanedInProgress as “may still be answerable”). The frontend (useAgentQuestions.ts’s SAFE_TO_RETRY_VIA_FOLLOWUP allowlist) matches on this error’s text (the “no pending AskUserQuestion” prefix) to redeliver as a follow-up message instead of rolling back — keep that exact prefix stable if this message ever changes. See docs/reports/REPORT_WORKING_STATE_REGRESSION_AND_STUCK_QUESTION_PANEL_2026_07_27.md §2.7/§2.8.

pub fn deny_question( &self, tool_use_id: String, message: String, ) -> Result<(), String>

Decline a parked AskUserQuestion via the Agent SDK control protocol — the Cancel button / Escape in AgentQuestionPanel.tsx. Structurally a mirror of answer_question (same pending_questions lookup/removal, same dead-air safety net — the CLI can abandon a pending tool_use whose turn already ended regardless of whether the response was an allow or a deny), but sends behavior: "deny" with a message instead of behavior: "allow" with updatedInput. This is a general, documented Agent SDK mechanism (PermissionResult deny case for the canUseTool callback) — AskUserQuestion goes through the exact same callback as ordinary tool permission requests, confirmed against the official Agent SDK docs (code.claude.com/docs/en/agent-sdk/user-input). Spec: docs/specs/SPEC_AGENT_CONTROL_PROTOCOL_2026_06_15.md.

See answer_question’s doc comment for the pending_questions in-memory-only caveat and the exact error-prefix stability requirement (useAgentQuestions.ts’s SAFE_TO_RETRY_VIA_FOLLOWUP allowlist matches on this method’s error text too, since it shares the identical lookup).

fn reply_to_pending_question( &self, tool_use_id: String, not_running_error: &'static str, response_body: impl FnOnce(Value, &str) -> Value, resume_msg: String, logs: &'static DeadAirLogs, ) -> Result<(), String>

Removes the parked AskUserQuestion tool_use_id and replies to it with response_body(questions, tool_use_id) via Self::send_control_response_with_fallback. The “no pending AskUserQuestion” error prefix is matched by the frontend.

fn send_control_response_with_fallback( &self, tx: Sender<String>, request_id: String, response_body: Value, tool_use_id: String, resume_msg: String, logs: &'static DeadAirLogs, ) -> Result<(), String>

Sends the control_response for a parked can_use_tool request and arms the dead-air fallback, which re-delivers resume_msg as a follow-up user turn if no stdout activity follows the response.

pub fn decide_tool_permission( &self, tool_use_id: String, outcome: &str, feedback: Option<String>, ) -> Result<(), String>

Decide a parked ordinary tool-permission request via the Agent SDK control protocol — the eventual AgentDecisionPanel Allow/Deny buttons, once should_route_to_decision_panel is flipped on (Phase 2, SPEC_DECISION_PROMPT_2026_04_24.md / SPEC_AGENT_CONTROL_PROTOCOL_2026_06_15.md §5). Mirrors answer_question/deny_question exactly — same pending_* map lookup/removal, same dead-air safety net — combined into one method because the frontend’s tool:decision RPC already carries outcome as a single field rather than two separate commands.

outcome must be "allow" or "deny" — validated by the caller (websocket.rs’s tooldecision handler) before this is reached, the same division of responsibility as that handler’s existing scope validation. An unrecognized value is treated as "deny" (fail closed) rather than panicking or silently allowing.

On allow, updatedInput echoes the ORIGINAL input verbatim — this method does not support editing the call before approving it (no UI for that exists; SPEC_DECISION_PROMPT_2026_04_24.md never scoped one). On deny, feedback becomes the CLI-facing message (SPEC_DECISION_PROMPT’s G6: “Denials carry user-typed feedback verbatim to the agent”); a caller with no feedback gets a generic default so the model still learns the call was refused.

See answer_question’s doc comment for the pending_permissions in-memory-only caveat (a fresh controller instance — pane reopen, any process respawn — starts with an empty map) and for why the error text names the tool_use_id and the likely cause.

pub(super) fn push_stdin(inner: &Arc<Mutex<PersistentInner>>, line: String)

Push a raw NDJSON line to the live stdin (used to emit control_responses from the stdout-reader task, which only holds an Arc<Mutex<Inner>>).

pub(super) fn handle_control_frame( kind: &str, parsed: &Value, block_id: &str, inner: &Arc<Mutex<PersistentInner>>, )

Handle a control-protocol frame from the CLI’s stdout. control_request of subtype can_use_tool: AskUserQuestion is parked (the frontend panel — rendered from the assistant stream — answers it via answer_question); every other tool is routed to should_route_to_decision_panel, which today always says no, so it is auto-allowed to preserve the current bypass/yolo UX (Phase 1; see that function’s own doc comment for why Phase 2, #551, isn’t simply “flip it to true”). control_response frames (replies to requests we initiate, none today) are logged and dropped. These frames are NOT conversation output and never reach the blockfile. Spec: docs/specs/SPEC_AGENT_CONTROL_PROTOCOL_2026_06_15.md §4.2.

impl PersistentSubprocessController

pub fn stop_process(&self, force: bool) -> Result<(), String>

pub(super) fn session_held_elsewhere(&self, sid: &str) -> Option<(String, bool)>

Another process already running on session sid: (block id, closing). Checks live persistent controllers in the registry, then blocks mid-close (out of the registry, process not yet exited).

pub(super) fn agent_held_by_other_pane(&self, uid: &str) -> Option<String>

Another pane of this process holding agent uid’s single-live-instance lease: its block id. Whatever session either pane holds — a pane with a stale id refused because the agent’s live pane holds the lease is still “open in another pane”, not “another instance” (Camper on #3935).

pub(super) fn request_stop(&self, request: KillRequest) -> Result<(), String>

pub(super) fn request_stop_on( inner_arc: &Arc<Mutex<PersistentInner>>, request: KillRequest, ) -> Result<(), String>

[request_stop] over just the shared state, for the 'static future Controller::shutdown returns.

pub(super) fn stop_for_config_restart(&self)

Apply a deferred runtime-config restart: stop the current process (as stop_process(false) does) WITHOUT clearing the restart token the turn boundary set (PersistentInner::config_restart_generation), so the exit handler brings the replacement up. It never sets the token either: a Stop that landed between the boundary and this call already cleared it and took the kill channel, and must stay in force (codex P1 on #3990).

pub(super) fn request_stop_draining_deferred( &self, request: KillRequest, ) -> Vec<String>

A kill for teardown: drains the deferred queue in the SAME inner acquisition that issues the kill, and returns what it drained for the caller to report. Draining in a second acquisition left a window after the kill request where a flush could write a deferred message into the process being killed, and leave the report an empty queue: lost, with no stranded warning (codex P2 on #3562). Once this lock is released, stop_pending holds anything queued later.

pub(super) fn request_stop_inner( inner_arc: &Arc<Mutex<PersistentInner>>, request: KillRequest, drain_deferred: bool, config_restart: bool, ) -> Vec<String>

config_restart: this stop is a deferred runtime-config restart’s own (Self::stop_for_config_restart), which leaves the restart token as the boundary left it. Every other stop clears it in this same acquisition, so a Stop or pane close landing after the restart was committed wins, and nothing respawns behind it.

pub fn session_id(&self) -> Option<String>

pub fn needs_spawn(&self) -> bool

True when this controller is registered but has no live process and no spawn already in flight — the state a lazily-registered controller sits in between register (“spawns on first message”) and its first send_message.

In that state inject_message cannot deliver: it has no spawn config and only writes to a live stdin_tx, so it returns “persistent process not running”. The caller can, however, start a turn — run_agent_turn re-reads the spawn config from block metadata and calls send_message, which does spawn. This predicate is what lets the reactive delivery path tell those two situations apart. See docs/reports/REPORT_JEKT_DELIVERY_DROPS_UNSPAWNED_PERSISTENT_AGENTS_2026_09_03.md.

The invariant this must uphold: needs_spawn() == true implies a subsequent send_message takes decide_send_action’s BecomeSpawner branch — i.e. it really spawns and delivers. It must never be true in a state where send_message would instead return SendAction::Queued, because Queued returns Ok(()) having delivered nothing, the reactive path maps that to Ok(true), and the caller is told the message landed when it is still sitting in the queue. cloud_subscriber only retries on !success, so a false success is a permanently lost message.

That is why all three exclusions are here, and each mirrors one of decide_send_action’s own guards:

• stdin_tx.is_none() — a live process is steerable; inject_message handles it and no turn start is wanted.
• !spawning_in_progress — a caller has already claimed this spawn round (see that field’s doc comment on the concurrent-spawn TOCTOU race). Reporting “needs spawn” here would invite a second, racing spawn and orphan a child.
• !drain_claim — a RetryFlush drain owns the round. This one is subtle and was missed on the first cut (reagent P1 on PR #2960): when a drain’s target process dies mid-flush, the drain deliberately retains its claim for the fallback respawn while the exit handler clears stdin_tx. Without this term, that window reports “needs spawn”, decide_send_action then returns Queued on the still-held claim, and the message is silently queued while the caller is told it was delivered.

Both excluded windows are retryable rather than lost: the caller gets the original delivery error back, which is what lets it come again.

Racy by nature, and safe to be: the process can exit the instant after this returns false. It is a routing hint, not a guarantee — the spawn claim inside send_message is what actually serialises spawners. What it must not do is report true for a state that cannot spawn.

pub fn request_restart_when_idle(&self) -> bool

Ask this controller to restart itself once the current turn ends, instead of being torn down and replaced right now.

Returns true when the restart was deferred (a turn is in flight, so the caller must NOT replace the controller), false when the pane is idle and an immediate replace is safe — the caller then proceeds exactly as before.

Called from resync_controller’s forced-replace path. See PersistentInner::restart_when_idle for what the old kill-immediately behaviour destroyed.

Deliberately keyed on the health monitor’s is_active_turn rather than on stdin_tx.is_some(): a live process between turns is idle and should be replaced immediately (that’s the common /model case, and deferring it would leave the change unapplied until the next turn happened to end). It’s specifically an IN-FLIGHT turn that must not be interrupted.

impl PersistentSubprocessController

The MCP self-heal’s steps for this pane (backend::mcp_health decides them; docs/specs/SPEC_AGENT_MCP_SELF_HEAL_2026_10_10.md).

pub(super) fn observe_mcp_frame(&self, frame: &Value, generation: u64)

A frame from process generation: the agentmux MCP server’s status (system/init, our mcp_status) or a reconnect’s outcome. Anything else, or a frame from a replaced process, is ignored.

pub(crate) fn mcp_heal_now(&self) -> Result<(), &'static str>

agentmux-mcp doctor --heal: start the repair over now. Err says why nothing started (mcp_health::heal_now).

fn mcp_process_key(&self, generation: u64) -> String

One CLI process of this controller: a replacement controller restarts its generations, so the generation alone could name an older process.

fn carry_out_mcp_step(&self, decision: Decision, generation: u64)

fn send_mcp_request(&self, line: String, after: Duration, generation: u64)

Write a control request to process generation, after after; not if it has been replaced by then.

fn restart_now_or_after_turn(&self)

Restart the CLI (the same session) with the config now in force, which for a pane in the safe profile means without the shell prefix: at once when idle, else when the turn ends, through the deferred runtime-config restart (turn_boundary_locked, respawn_after_config_restart). The turn check and the flag share one acquisition, as in Self::request_restart_when_idle.

fn show_mcp_notice(&self, notice: &Notice)

fn tell_agent_mcp_is_down(&self)

The [AgentMux] note that AgentMux gave up, so the agent tells the operator and knows its tools are gone. By UID when the pane has one.

impl PersistentSubprocessController

pub(super) async fn run_process_waiter(ctx: ProcessWaiterCtx)

Body of the process waiter task.

impl PersistentSubprocessController

pub(super) fn decide_send_action_from( &self, json_str: &str, skip_if_seq_queued: Option<u64>, origin: Option<TurnInput>, ) -> SendAction

[Self::decide_send_action], saying who the input is from (turn provenance, SPEC_AGENT_SELF_QUIT §6.3). A direct delivery reports it with the turn mark; a queued one tells the tracker about the turn it will start.

pub(super) fn deliver_direct( &self, json_str: &str, was_active: bool, ) -> Result<(), String>

The write half of SendAction::DeliverDirect, under a second inner acquisition. On failure nothing was written, and the turn the decision reserved is handed back: left active, it would hold every deferred message for a result that never comes. Nobody else can have started a turn meanwhile, since they all saw it reserved and deferred, and the watchdog each armed releases them once it reads idle.

Written through Self::try_write_stdin_locked, like every other write site, so the gates are re-checked under THIS acquisition: a stop or config restart committed after the decision (heartbeat and status publish run in between, with stdin_tx still live) refuses the write instead of putting the message into the dying process (reagent P1 on #3562). The same call tracks it into the current generation’s retry batch (codex P1 on PR #3513: eager resume can reach DeliverDirect with its --resume unconfirmed and no seed tracked), reading the generation in this same acquisition (reagentx P1 on PR #2373).

pub(super) fn release_spawn_claim_and_drain_queue( &self, spawn_succeeded: bool, retry_config: PersistentSpawnConfig, own_seq: u64, )

Releases the exclusive spawn claim taken by decide_send_action returning BecomeSpawner. spawn_succeeded distinguishes two very different situations:

• Failed spawn: discards only the entry with own_seq — the specific message THIS spawner pushed when it claimed BecomeSpawner (SendAction::BecomeSpawner::own_seq), found by queue identity rather than assumed to be at the front. codex P2 on PR #2360 (round 14, commit 8c2bc99ab): the queue is NOT always empty at the moment a new spawner claims it — the “second stall” path (drain_queue_after_successful_spawn with allow_fallback_respawn: false) deliberately releases spawning_in_progress while leaving genuinely leftover messages queued (see that function’s own doc comment). A later send_message can then claim BecomeSpawner and push_back its own message BEHIND those leftovers. Assuming “front == my own message” in that case discarded an OLDER, unrelated, already- accepted prompt instead of the actually-failed one — silent data loss, plus handing the wrong (already-failed) message to the fallback respawn. Originally fixed by matching content instead of position, which still left two GENUINELY DIFFERENT messages sharing identical text (e.g. two “yes” replies) ambiguous; seq matching (issue #2365) closes that residue too. codex P1 on PR #2360 (sixth review pass): leaving this message queued let an unrelated LATER successful spawn silently execute a prompt the caller was already told had failed (send_message/ retry_after_resume_failure already report the failure), sometimes duplicating a message the user had re-sent by hand. Anything else queued (from other callers who got SendAction::Queued and were already told “accepted”) is left in place for the next successful spawn.
• Successful spawn: hands the drain off to a background task that delivers everything queued, in order, via Sender::send (which awaits free capacity) rather than try_send. codex P2 on PR #2360 (sixth review pass): a synchronous try_send loop popped a message off the queue and then discarded it outright if the bounded stdin channel was momentarily full — losing input despite already having told that caller “accepted”. Deferring to a task lets delivery simply wait for capacity instead.

If the background task discovers the process it was meant to drain into has ALREADY died (stdin_tx gone) with messages still left queued, it hands off to respawn_once_for_leftover_queue using retry_config — reagentx/codex P1 on PR #2360 (sixth review pass, rounds 3-4): a fast-dying child can let the process-waiter run this SAME controller’s own retry_after_resume_failure (via decide_send_action returning Queued, since this spawn’s claim hasn’t been released yet) BEFORE this caller’s own thread even reaches this function; that retry’s Queued branch then does nothing further, assuming (wrongly, in this exact race) that this drain will deliver it. Without a fallback respawn, NOTHING is ever left responsible for the leftover messages or for telling the frontend the turn ended — the ORIGINAL exit deliberately suppressed its own terminal-status publish expecting the retry to eventually publish one.

pub(super) fn drain_queue_after_successful_spawn( &self, retry_config: PersistentSpawnConfig, allow_fallback_respawn: bool, )

Drains the queue via a background task after a successful spawn — see release_spawn_claim_and_drain_queue’s doc comment for the try_send → Sender::send rationale. allow_fallback_respawn bounds retry depth to exactly one extra hop: true from the public entry point, false when called from respawn_once_for_leftover_queue itself, so a SECOND stall just publishes a status update instead of cascading indefinitely.

pub(super) fn drain_queue_with_claim( &self, retry_config: PersistentSpawnConfig, allow_fallback_respawn: bool, claim: QueueDrainClaim, )

The queue-drain loop itself, parameterized by which exclusivity claim it runs under (QueueDrainClaim) — the post-spawn drain (SpawnClaim) and issue #2367’s retry-batch flush (RetryFlush) share every delivery invariant (Sender::send backpressure, seed-aware retry-batch appends, delivery-order persistence, stall handling); the ONLY difference is which flag they release. RetryFlush callers always pass allow_fallback_respawn: false: the flush targets an already-running process, so a stall means that process died — leftovers stay queued for the next spawn and the stalled branch publishes a status update.

pub(super) fn clear_session_id_for_fresh_spawn(&self)

Attempts exactly one fallback respawn when either call site above is about to release its claim with messages still queued and nobody left responsible for delivering them. Forces session_id empty so this fallback spawn never attempts --resume — reusing a possibly-still-stale session id here would risk repeating the exact failure this whole retry mechanism exists to recover from. If THIS spawn also fails, or its own process also dies before its own drain completes, gives up and publishes a status update rather than cascading indefinitely — the queue itself is never discarded (this function never pops anything itself — it isn’t tied to a specific message the way the original spawn attempt was), so a genuinely later, unrelated send will still eventually pick it up. Atomically clears the ambient session id AND reserves a fresh spawn generation, so every reader task belonging to any EXISTING generation is stale from this point on. Used by both fresh-start respawn paths (respawn_once_for_leftover_queue, retry_after_resume_failure’s BecomeSpawner arm) in place of a bare session_id = None.

codex P1 on PR #2500 (second round): clearing alone left a window — until spawn_process’s own generation bump, which happens AFTER the --resume decision reads session_id and after the process is spawned — where the dying generation still equaled spawn_generation, so its stdout reader’s stale-sid echo passed try_capture_session_id’s currency gate (issue #2366) and was re-adopted: the supposedly fresh spawn could reattach --resume <stale-sid> with no retry payload armed to catch the repeat failure. Reserving the next generation in the same lock acquisition as the clear closes the whole window.

The reserved generation is never itself spawned (spawn_process bumps again) — see spawn_generation’s doc comment for why the gap is inert. A stop_process racing into the reserve window records a StopRequested for the never-spawned generation, which the resume state machine ignores; both callers only reach this point after the prior generation’s tracking has already resolved, so no stop-intent is lost that wasn’t equally lost before.

pub(super) fn respawn_once_for_leftover_queue( &self, config: PersistentSpawnConfig, )

pub(super) fn emit_message_accepted(&self, message_id: Option<&str>)

Send a user message to the running CLI process. If the process isn’t spawned yet, spawns it first. Emit agent-message-accepted for a given message_id, if set. Mirrors the subprocess controller’s emit_message_accepted — signals the frontend to promote the pending entry from queued to in-document.

pub(super) fn persist_message_to_blockfile(&self, json_str: &str)

Persists a formatted stdin JSON line to the blockfile + global zone so parseHistoryLines can reconstruct the user_message node on the next pane open. Published as an echo: "stdin" transcript event (Phase 5a-3c): the pane already shows the message through the agent-message-accepted path (UUID node), so it doesn’t add a second node, but its stream keeps no gap where this record sits.

pub fn send_message( &self, message: String, config: PersistentSpawnConfig, ) -> Result<(), String>

pub fn send_message_from( &self, message: String, config: PersistentSpawnConfig, origin: Option<TurnInput>, ) -> Result<(), String>

Self::send_message, saying who the message is from, so the turn it starts (or joins) carries that provenance (SPEC_AGENT_SELF_QUIT §6.3). None = unknown, which never counts as the user.

pub fn send_message_with_images_from( &self, message: String, images: Vec<Value>, config: PersistentSpawnConfig, origin: Option<TurnInput>, ) -> Result<(), String>

Self::send_message_from with Anthropic image content blocks placed before the text (image attachments, SPEC_AGENT_PANE_IMAGE_ATTACHMENTS_2026_09_26.md §6.6). With no images the line is byte-for-byte what send_message_from always wrote. Stored copies drop the images again — see persist_message_to_blockfile.

impl PersistentSubprocessController

pub(super) fn flush_error_line_now(&self, line: String)

Appends a terminal error-result line to this block’s output now and surfaces its classified failure: a held-back line confirmed final outside the process-waiter task (its retry failed to launch, or its generation was superseded), or an error frame synthesized for prompts that could not be delivered. See retry_after_resume_failure’s doc comment for when a held line is flushed rather than dropped.

pub(super) fn emit_fresh_outcome_now( &self, attempted_sid: String, continued: bool, )

A Fresh outcome whose new session will carry AgentMux’s record of the conversation, so the pane says “continued” rather than “new”.

pub(super) fn fresh_spawn_would_continue(&self) -> bool

Whether a fresh spawn right now would carry a continuation packet (what spawn gives any no---resume spawn onto prior history). Reads the transcript tail: call it on rare paths, never under inner.

fn emit_session_outcome_now_with( &self, outcome: SessionOutcome, attempted_sid: String, actual_sid: Option<String>, continued: bool, )

pub(super) fn has_prior_transcript(&self) -> bool

Does a transcript already exist for this pane — either in this channel’s own blockfile, or in the agent’s GLOBAL transcript zone?

The second half is the one that matters for fresh_start_needs_disclosure: on a cross-channel or cross-version open this channel’s blockfile is empty, but the pane still renders the full prior conversation through app_api::global_output_source’s read fallback. Mirrors that function’s checks (agent-anchored, not archived, non-empty output) so “the pane will show history” and “we disclose a fresh start” can’t disagree.

Two stat calls at most, on the spawn path only — cheap enough to run unconditionally behind the caller’s own generation gate.

pub(super) fn find_continuation_session_id( &self, config: &PersistentSpawnConfig, ) -> Option<String>

The session to continue on a first spawn that holds no id of its own (SPEC_DURABLE_CONVERSATION_MEMORY_2026_09_23.md §5 P0a): the session of the history this pane renders (pane_history_session_id), when --resume would find it under this spawn’s config dir and it isn’t already poisoned.

pub(super) fn continuation_packet(&self) -> Option<String>

AgentMux’s record of the conversation this pane renders, as a continuation packet for a process that starts without it (SPEC_DURABLE_CONVERSATION_MEMORY_2026_09_23.md §4.4).

pub(super) fn carry_continuation(&self) -> Option<String>

The continuation packet a fresh spawn onto prior history carries, or None when there is no record to give it.

Carrying one also retracts the “Couldn’t resume the previous conversation — started a new one” banner a rejected --resume raised on the previous generation: the conversation wasn’t lost, this session continues it from AgentMux’s record. Same reasoning as the Resumed retract in the stdout reader — the banner must not contradict the “continued” outcome the pane shows.

pub(super) fn find_recovery_session_id( &self, config: &PersistentSpawnConfig, ) -> Option<String>

After a confirmed-stale --resume failure, try to recover a REAL session instead of giving up and starting blank (docs/status/STATUS_CROSS_CHANNEL_RESUME_STALE_SESSION_ID_2026_08_20.md). Looks for the largest on-disk session under this spawn’s own CLAUDE_CONFIG_DIR/working dir — the same “largest session wins” recovery session_backfill::backfill_session_ids already trusts for the analogous cross-channel-open case.

Returns None (caller falls back to starting blank, same as before this existed) when: CLAUDE_CONFIG_DIR isn’t set on this config, no session exists there, or the only candidate found is the SAME id already confirmed poisoned — guards against looping forever “recovering” an id that is itself dead (nothing on disk changes between attempts, so an unguarded retry would rediscover the identical bad id every time).

pub(super) fn retry_after_resume_failure( &self, retry_generation: u64, config: PersistentSpawnConfig, entries: Vec<QueuedRetryEntry>, held_error_line: Option<String>, attempted_sid: String, )

Retries the message that triggered a --resume <sid> attempt this controller’s own stderr reader just confirmed is unreachable (“No conversation found with session ID” — see poison_resume). Called from the process-waiter task once the doomed process has actually exited, via the weak self-reference (mirrors SubprocessController’s queued-message drain — see set_self_ref), and ONLY for a persistent_resume::ResumeEffect::FireRetry (a ConfirmedRetry generation’s exit) — never for an unrelated exit.

Respawns with the stale session_id replaced: by a recovered on-disk session if find_recovery_session_id finds one, otherwise cleared so no --resume is attempted. Redelivers EVERY message the doomed process’s stdin channel had accepted (the drain appends each later delivery to the batch — persistent_resume::ResumeEvent::MessageAppendedToRetryBatch — so this is a batch, not just the one that triggered the spawn) — does NOT re-persist to the blockfile or re-emit agent-message-accepted for any of them, since both already happened correctly on each message’s original (failed) attempt; only the underlying CLI process needed a fresh start.

This is itself a spawn attempt, and must not race a genuinely concurrent send_message call the same way the ORIGINAL doomed spawn could — see PersistentInner::spawning_in_progress’s doc comment. By the time this runs, the original send_message call that triggered the doomed process has long since returned (its own spawn-claim-and-deliver sequence completed synchronously, well before this process even exited), so the WHOLE batch can safely go through decide_retry_batch_action — the SAME decision send_message uses, but enqueueing everything atomically in one lock acquisition (see its own doc comment for why per-message decisions aren’t safe for a batch).

A held-back error line must reach the user if a confirmed retry turns out NOT to actually launch (#2371) — otherwise an already-accepted prompt ends in total silence: neither the original error nor a replacement one. It is flushed only when a BecomeSpawner respawn fails; the FlushClaimed and Queued arms hand off to a background drain whose own stalled_with_leftovers branch already publishes a status update on genuine total failure, so those arms drop the line instead — see the Queued arm below for why flushing eagerly there would reproduce the very bubble #2371 removed.

retry_generation is the spawn generation whose ProcessExited fired this retry (the process-waiter’s own my_generation_wait) — decide_retry_batch_action needs it to tell a live NEWER spawn apart from the impossible “our own process is somehow still running” case (issue #2367). attempted_sid is the id that was just confirmed unreachable — threaded through from persistent_resume::ResumeEffect::FireRetry so this function can emit the eventual EmitSessionOutcome itself once recovery is known (reagentx + Codex on PR #2693 — persistent_resume.rs can no longer decide Fresh-vs-Resumed at the point it fires this retry, since recovery might succeed).

pub(super) fn decide_retry_batch_action( &self, retry_generation: u64, first: &QueuedRetryEntry, rest: &[QueuedRetryEntry], ) -> RetryBatchAction

Same shape of decision as decide_send_action, but atomically enqueues the ENTIRE batch (not just one message) in every arm — used only by retry_after_resume_failure. codex P2 on PR #2360 (sixth review pass, round 6): deciding and enqueueing a multi-message retry batch one call at a time (each through its own decide_send_action call) left a window between them where a genuinely new, unrelated message could interleave into the MIDDLE of the same original batch, reordering it relative to how the doomed process actually received it. Batch prepend/dedup rules live in Self::prepend_retry_batch.

Issue #2367 (spec §4, option 2): the live-process outcome is no longer a caller-side try_send (DeliverDirect) — that let a concurrent send race ahead of, or into the middle of, the batch. It is now RetryBatchAction::FlushClaimed: batch prepended and drain_claim taken under this one lock, then flushed through the shared queue drain.

pub(super) fn prepend_retry_batch( inner: &mut PersistentInner, first: &QueuedRetryEntry, rest: &[QueuedRetryEntry], )

Prepends a retry batch to pending_send_messages, deduping only first by seq. codex P2 on PR #2360 (sixth review pass, round 11): prepend rather than append — this batch represents messages accepted by the doomed process BEFORE whatever’s currently queued (anything queued arrived after the original spawn claimed spawning_in_progress, so it’s chronologically later); appending would let the fresh process receive later-arriving input first.

first may itself STILL be sitting in the queue (see decide_send_action’s doc comment on skip_if_seq_queued) — always at index 0 if so, since it’s the ONLY thing ever present when a claim starts and nothing but push_back ever touches this queue elsewhere. In that case rest belongs immediately after it (same batch, preserving order), not ahead of it. rest is always pushed as-is: dedup must never apply WITHIN the same batch (two entries can legitimately carry identical text and both need redelivering).

impl PersistentSubprocessController

pub(super) fn settle_empty_resume_retry( &self, retry_generation: u64, config: PersistentSpawnConfig, held_error_line: Option<String>, attempted_sid: String, )

retry_after_resume_failure’s handling of an EMPTY retry batch: --resume <sid> was attempted with nothing queued — the eager-resume path (issue #3463) — and the CLI rejected the id before any prompt arrived. There is no message to re-send, so this is a terminal idle-resume failure, not a retry. Codex P1 on PR #3538: returning early as the old empty-batch branch did left the pane with no controller status ever published, because the process-waiter hands FireRetry (not PublishDone) to the retry path on the assumption that a launch follows. Complete the recovery decision without a launch:

• adopt whatever find_recovery_session_id found (or None) so the NEXT message resumes it through the same gated, tracked --resume a first-time resume gets — its outcome (Resumed) is emitted then, by the CLI’s actual confirmation, never claimed here (codex P1 on PR #2693); with no candidate, Fresh IS decidable now and is disclosed now;
• publish the same terminal status a PublishDone exit would.

No “Reconnecting…” is shown or resolved here, unlike the non-empty path: no user turn is in flight, so there is no gap to narrate.

EVERYTHING here is conditioned on this retry’s generation still owning the controller state (codex P1 on PR #3551, both rounds): a message that arrived after the doomed process cleared stdin_tx but before this ran may already have spawned a newer generation with its own live session id — possibly resuming a DIFFERENT conversation. Overwriting that id, stamping done over it, or appending a Fresh disclosure to its history would each be wrong. The check runs twice: once before the recovery search (which does disk I/O outside the lock, and must not even start for a superseded generation), and again under the same lock acquisition as the mutation, so nothing that raced in during the search can be overwritten either. spawn_generation is bumped in the same lock acquisition that installs a new spawn, and spawning_in_progress covers the claim window before it — together they say whether anyone has moved on.

The side effects (outcome frame, status publish, error flush) run UNDER that same lock acquisition, not after it (reagent P1 and codex P1 on PR #3551, rounds three and four). Every scheme that released the lock first had a window: a check-then-act with nothing holding the state still, or a spawn claim that either this settlement took (fine) or someone else already held (the eager path’s own unwinding claim — whose owner could release it mid-effect and let a send become a newer generation while the Fresh line was still being written). Holding inner across the effects makes the window not exist: a concurrent decide_send_action blocks on the lock and, once it gets it, becomes a spawner on the already-adopted session. This is safe because nothing below re-enters the controller — the broker and event bus are mpsc channel sends, the block-file append and failure persistence touch the file store and SQLite only — and cheap enough for a path taken once per failed eager resume. It is the one deliberate exception to the file’s convention of dropping inner before effects, which exists because most effect handlers call back into methods that lock it; these three do not.

impl PersistentSubprocessController

pub(super) fn lease_then_gate( &self, config: &PersistentSpawnConfig, gfs: Option<&FileStore>, history_session: Option<String>, ) -> Result<Option<Arc<HeldAgentLease>>, String>

Claims this agent’s single-live-instance lease, then runs the resume gate (apply_resume_gate_with) under it.

The lease is one live instance per agent across every AgentMux instance on this host (SPEC_AGENT_SINGLE_LIVE_INSTANCE_2026_09_24.md Phase 1), held for the spawned process’s lifetime. Claiming it FIRST means the gate’s sweep, relocation and settle (spec §4.3) run for one spawn of an agent at a time: two controllers of one agent racing those steps kept opening narrower windows (#3924). The session is only a hint to the lease — recorded, not a key — so the gate settling the final id afterwards changes nothing about the claim.

When leasing is unavailable (no store, or no agent UID) the gate runs unguarded, as it did before Phase 1.

pub(super) fn apply_resume_gate_with( &self, config: &PersistentSpawnConfig, gfs: Option<&FileStore>, history_session: Option<String>, )

The one resume gate (SPEC_RESUME_GATE_AND_SAME_IDENTITY_CONTINUATION_2026_09_25.md §4.2): before this spawn passes --resume, only the head of the agent’s segment chain may be the session it resumes. A stale id — the conversation moved on in another channel, build or pane — is redirected to the head when this config dir reaches it, and cleared otherwise, so the spawn goes fresh and carries AgentMux’s record. Runs after every path that sets the id (hydrate from config, first spawn onto rendered history) and under the agent’s lease (lease_then_gate), before the held-elsewhere check, which then applies to whatever id survives. Never fails the spawn: anything unknown leaves the id as it was.

history_session: on a first spawn holding no id, the session of the history the pane renders when --resume can’t reach it here. It is a candidate for relocation only (a rebuild onto a new login of the same identity); any other decision leaves the spawn holding no id, as before.

pub(super) fn reconcile_memory_before_spawn( &self, config: &PersistentSpawnConfig, )

Reconcile the agent’s memory record with the folder this spawn is about to use, before the provider starts, within memory_reconcile::RECONCILE_BUDGET (SPEC_MEMORY_FOLLOWS_THE_AGENT_2026_09_24.md §2.1.3). Never fails the spawn. Off for an agent with agent:memoryrecord = off.

pub(super) fn record_segment_start( &self, agent_uid: &str, config: &PersistentSpawnConfig, attempted_resume_sid: Option<&str>, forked_from: Option<&str>, relocated: bool, carries_packet: bool, lease_epoch: Option<u64>, ) -> Option<(String, String)>

Records the start of the segment this spawn begins. None when the spawn carries no agent UID (quick-launch panes, a continuation that resumes before its row exists) or the write fails.

pub(super) fn segment_start( &self, agent_uid: &str, config: &PersistentSpawnConfig, attempted_resume_sid: Option<&str>, forked_from: Option<&str>, relocated: bool, carries_packet: bool, lease_epoch: Option<u64>, ) -> Start

The Start event record_segment_start writes. A relocated fork records no session until it reports its own: the attempted id names the copy, which is swept if the process dies first, so as the head it would send the next spawn to --resume a file that is gone instead of relocating again (spec §4.3).

impl PersistentSubprocessController

pub(super) fn spawn_process( &self, config: PersistentSpawnConfig, resume_retry_payload: Option<QueuedRetryEntry>, ) -> Result<(), String>

Spawn the persistent CLI process. Called only while the caller holds the exclusive spawn claim (spawning_in_progress, see decide_send_action) — never directly.

impl PersistentSubprocessController

pub(super) fn set_status(inner: &mut PersistentInner, status: &str)

pub(super) fn get_status_snapshot(&self) -> BlockControllerRuntimeStatus

pub(super) fn publish_status(&self)

pub(super) fn publish_runtime(&self)

Announce only what the process runs (or that none does), without repeating the controller status. For the paths that move the status off “running” without publishing it themselves (stop).

pub(super) fn spawn_status_heartbeat(&self)

Periodic (low-frequency) republish of the current controllerstatus while a turn is active — a self-healing backstop independent of publish_controller_status’s persist: 1 (which only helps a reconnecting subscriber) and the frontend’s focus-triggered reconcile (which only fires on a background→foreground transition). If a single live push is missed for some other reason (e.g. a throttled/backgrounded renderer coalescing WS messages) while the window stays foregrounded and connected the whole time, nothing else corrects it until the turn actually ends. This shrinks that window from “forever” to at most one heartbeat interval. HEARTBEAT_SECS is well below the frontend’s own STUCK_THRESHOLD_MS (45s, diagnostic-only) and LIVENESS_RECOVERY_MS (180s, force-recovery) so a missed push self-heals long before either of those fire. See REPORT_LOGIN_PERSIST_FAILURE_AND_STUCK_WORKING_2026_07_27.md §4 item 5. Builds the snapshot with agent_runtime_status rather than get_status_snapshot, since the spawned task only holds cloned Arcs, not &self.

A latent duplicate-loop race is an existing, already-accepted contract (reagent P2 on the PR that introduced this function): if a turn ends and a new one starts again within one HEARTBEAT_SECS window, the old loop hasn’t yet woken up to observe is_active_turn() == false and break, so both the old and new loop can run concurrently for that window. Harmless — publish_status republishing the same (or a slightly stale) snapshot twice is idempotent from the frontend’s point of view — so this is accepted rather than fixed, matching the pre-existing pattern.

pub(super) fn spawn_status_heartbeat_with_interval( &self, heartbeat_interval: Duration, )

Interval parameterized out of spawn_status_heartbeat so a test can drive it with a short interval instead of waiting out HEARTBEAT_SECS.

pub(super) fn cli_started_pass(&self)

The CLI started a pass by itself and the tracker has marked it active (mark_turn_active_from_cli): publish the flip and keep the status heartbeat going for it, as for a pass srv started.

pub(super) fn mark_turn_active_and_publish(&self)

Marks a turn active (re-arming the status heartbeat only if it was previously idle) and publishes the resulting status flip. Shared by send_message and retry_after_resume_failure — both represent “a user message is about to be delivered,” just via different spawn paths. The heartbeat is re-armed conditionally: a mid-turn steering send already has one running, so re-spawning on every call would leak duplicate heartbeat tasks.

impl PersistentSubprocessController

pub(super) async fn run_stderr_reader(ctx: StderrReaderCtx)

Body of the stderr reader task; spawn_process keeps its JoinHandle for the process waiter.

impl PersistentSubprocessController

pub(super) async fn run_stdout_reader(ctx: StdoutReaderCtx)

Body of the stdout reader task; spawn_process keeps its JoinHandle for the process waiter.

impl PersistentSubprocessController

pub fn new( tab_id: String, block_id: String, broker: Option<Arc<Broker>>, event_bus: Option<Arc<EventBus>>, mstore: Option<Arc<Store>>, filestore: Option<Arc<FileStore>>, ) -> Self

pub fn with_identity_stores( self, id_store: Option<Arc<Store>>, identity_store: Option<Arc<Store>>, auth_key: String, ) -> Self

Supplies the dependencies start()’s eager-resume path needs (SPEC_PERSISTENT_CONTROLLER_EAGER_RESUME_ON_RECONNECT_2026_09_20.md): the Layer 3 identity/credential spawn gate’s two stores, and the auth key a spawned CLI needs for AGENTMUX_AUTH_KEY (MPS publish scoping via the bashwrap-invoked tool wrapper).

A separate builder step, not three more new() parameters, so every existing caller — the ~15 test constructions in this file among them — is unaffected. A controller built without this call simply cannot eager-resume even when agent:sessionid is present; it falls back to start()’s ordinary lazy “wait for the first message” path rather than spawning without the identity gate. That fallback is the deliberately safe default: eagerly resuming a possibly-deauthed agent’s session on whatever ambient credential happens to be present would silently reintroduce the exact vulnerability class SPEC_ACCOUNT_DELETE_DEAUTH_LAYERS_2_4_2026_07_14.md closed. See start()’s own doc comment for the full reasoning.

pub fn with_agent_lease_store( self, registry: Option<Arc<Registry>>, boot_id: Arc<str>, ) -> Self

Supplies the single-live-instance lease store (under the shared agent registry root) and this srv’s boot id. A separate builder step for the same reason as with_identity_stores. Without it the controller takes no lease — the pre-Phase-1 behaviour.

pub(super) fn acquire_agent_lease( &self, config: &PersistentSpawnConfig, session_hint: Option<&str>, ) -> Result<Option<Arc<HeldAgentLease>>, String>

Claim (or keep) this agent’s single-live-instance lease for the process spawn_process is about to start. Ok(None) when leasing is unavailable (no store) or the spawn env carries no agent UID — those run unguarded, as before Phase 1, and are logged. Err is the user-facing refusal: another instance runs this agent.

A lease this controller already holds (an earlier generation’s, still valid) is reused rather than re-claimed: a fresh handle for the same key would release the shared file when the old one dropped.

pub(crate) async fn check_admission( &self, uid: &str, agent: &str, ) -> Result<(), String>

The early admission check (spec I9) for this controller’s agent: refuses while another instance holds it. Runs before any shared write.

pub(super) fn fence_check(&self) -> Result<(), String>

The pre-turn fence (spec I4): Err if this pane’s CLI process lost its lease to another instance. Also stops that process, so it cannot keep driving the agent. Ok when no lease is held (nothing spawned yet — the spawn will claim — or leasing is unavailable).

pub fn release_to_other_instance(&self, deadline: Instant) -> bool

Hand this pane’s agent over to another AgentMux instance that asked for it (spec §4.6, POST /agentmux/agent/release): if this pane’s CLI process holds the agent’s single-live-instance lease, stop it gracefully — stdin EOF, so a turn in flight finishes, then kill at the deadline — and let its exit release the lease. false when this pane holds nothing (no process, or no lease).

fn stop_after_lease_loss( inner: &Arc<Mutex<PersistentInner>>, block_id: &str, why: &str, )

Kill the current CLI process after its lease was lost — the holder stops rather than racing the new one (spec §4.3).

pub fn set_self_ref(self: &Arc<Self>)

Sets the weak self-reference used by the process-waiter task to call back into retry_after_resume_failure once a doomed process (stale --resume session id) actually exits. Mirrors SubprocessController:: set_self_ref’s queued-message-drain pattern. Must be called by the caller right after wrapping a fresh instance in Arc — a controller that’s never had this called simply never retries (the check is a harmless no-op Weak::upgrade() failure), so this is safe to skip for e.g. throwaway test instances.

Trait Implementations

impl Controller for PersistentSubprocessController

fn start( &self, block_meta: HashMap<String, Value>, _rt_opts: Option<Value>, _force: bool, ) -> Result<(), String>

Lazy by default — “spawns on first message” — EXCEPT for a pane resuming an existing session across a backend restart/reconnect, per SPEC_PERSISTENT_CONTROLLER_EAGER_RESUME_ON_RECONNECT_2026_09_20.md. Without this, every persistent agent pane comes back from a crash/ OOM/update as an inert shell — correct layout, correct chrome, view: "agent", no live process — until a human or another agent happens to message it. Live incident: 3 of 7 panes in one window silently never respawned after 2026-09-20’s restarts (issue #3463).

agent:sessionid meta present and non-empty is the signal: this pane has a real, interrupted conversation rather than being genuinely new (which still spawns lazily, unchanged — no reason to burn a process before it’s needed). Mirrors ShellController::start()’s already- established “transparently revive on restart” behavior for shells.

Eager resume is gated on the SAME Layer 3 identity/credential spawn gate a live message send goes through (identity::resolver::inject_identity_env, SPEC_ACCOUNT_DELETE_DEAUTH_LAYERS_2_4_2026_07_14.md) — this is not an optional hardening pass, it is why this method does not simply spawn unconditionally the moment a session id is found. Skipping the gate here would let an agent whose bound account was deleted/revoked between the crash and the restart get silently respawned anyway, on whatever ambient credential happens to be lying around: exactly the vulnerability class that gate exists to close, reintroduced through a call site the gate was never wired into. Any of the following falls back to today’s lazy behavior — NOT a hard error, since a pane that merely stays inert is still recoverable by a message, unlike one spawned on the wrong credentials:

• id_store/identity_store/auth_key weren’t supplied (i.e. this controller was constructed without with_identity_stores() — should not happen for a production resync_controller call, but a test or future caller that omits it must not silently spawn ungated rather than just not eager-resuming).
• mstore is unavailable.
• cmd/cmd:args meta is missing (should not happen for a pane that has ever actually run — agent_open.rs always seeds them — but guessing CLI flags for a persistent agent is worse than not eager-resuming).
• The gate itself returns Err (SpawnGateError).

fn shutdown( &self, deadline: Instant, ) -> Pin<Box<dyn Future<Output = StopOutcome> + Send>>

Pane-close shutdown (spec §9.3), built on the Phase 0 measurements (§5.1): EOF alone does NOT end a running turn — the CLI finishes it first — but the control-protocol interrupt ends it in ~2s, and EOF then exits the process in ~0.5s. So: interrupt an active turn, wait for its result, EOF, and force-kill only at the deadline.

fn stop(&self, _graceful: bool, new_status: &str) -> Result<(), String>

Stop the controller. graceful waits for process to exit; new_status is the target state.

fn get_runtime_status(&self) -> BlockControllerRuntimeStatus

Get the current runtime status.

fn send_input( &self, input: BlockInputUnion, _seq: Option<u64>, ) -> Result<(), String>

Send input (terminal data, signal, or resize) to the controller. seq is the per-TermViewModel monotonic counter; None means fire-and-forget (no ordering).

fn turn_tracker(&self) -> Option<Arc<TurnActivityTracker>>

This controller’s turn tracker, for the turn ledger’s bookkeeping from outside the controller (a held message’s join hint). None for controllers that track no turns.

fn controller_type(&self) -> &str

Get the controller type (e.g., “shell”, “cmd”).

fn block_id(&self) -> &str

Get the block ID.

fn agent_id(&self) -> Option<String>

This block’s own live, spawn-time-captured jekt/muxbus identity, if it has one — an independent source of truth for the recipient- identity check in ReactiveHandler::inject_message_inner, deliberately NOT derived from ReactiveHandler’s own name_to_blocks/agent_info maps (checking a registry against itself would be a tautology and catch nothing). Default None for controller types that aren’t jekt-addressable at all (e.g. plain terminals) — only ShellController and PersistentSubprocessController currently override this, mirroring the two paths that already resolve a jekt-registration identity at spawn time (resolve_agent_id_for_jekt, muxbus_agent_id_from_env).

fn turn_provenance(&self) -> Option<TurnProvenance>

What started this agent’s current turn (SPEC_AGENT_SELF_QUIT §6.3), if the controller tracks it. None = unknown, which never counts as the user — the safe answer for controllers that can’t tell (ACP, App Server, subprocess: both origins reach them through one method).

fn set_agent_id(&self, id: Option<String>)

Refresh this block’s own captured jekt/muxbus identity (see agent_id’s doc comment). Called whenever ReactiveHandler::register_agent/register_agent_with_nonce (re-)registers THIS block’s block_id under a (possibly different) agent_id, so the two independently-written copies never drift apart (#2697: agent_id() was captured once at spawn and never refreshed, while the registry’s name bindings get re-keyed on every register_agent call — e.g. handle_reactive_register’s frontend-initiated HTTP path — causing a legitimately renamed or reconfigured agent’s own messages to be falsely rejected as an identity mismatch). Default no-op for controller types that don’t override agent_id either.

fn stable_agent_id(&self) -> Option<String>

This block’s STABLE jekt identity — AGENTMUX_AGENT_ID as captured once at spawn, never refreshed. Distinct from agent_id, which tracks the LIVE, renameable display name and is deliberately refreshed every turn (see that method’s doc comment on #2697). AGENTMUX_AGENT_ID is embedded verbatim in PR-body tags specifically because it does NOT change if the agent is later renamed — but ReactiveHandler’s registry only tracks one key per block, and every turn’s agent_id() refresh evicts whatever this block was previously registered under. ReactiveHandler::inject_message_inner’s recipient-identity check (#2695) uses this alongside agent_id() so a jekt addressed to the stable ID is validated correctly even after the primary registry key has moved on to the live display name. See INCIDENT_2026_09_09_JEKT_STABLE_ID_ALIAS.md. Default None, mirroring agent_id’s default — only PersistentSubprocessController currently overrides this.

fn stable_agent_uid(&self) -> Option<String>

Identity M2: this block’s UID (db_agents.id) as carried in its spawn env (AGENTMUX_AGENT_UID, set by build_persistent_spawn_env since M1a), captured once at spawn like [stable_agent_id]. Backs the registry’s UID confirmer: consulted only for targets that resolved by UID, and None — a registered-but-not-yet-spawned controller, or a block with no row — is “unverifiable”, not a mismatch. Default None; only PersistentSubprocessController overrides this.

fn as_any(&self) -> &dyn Any

Downcast support for concrete controller types.

fn stop_for_replace(&self, new_status: &str) -> Result<(), String>

Stop this controller because it’s being REPLACED by a new one for the same block (session restart / resync’s needs_replace path), NOT because the block itself is being closed. Must terminate this controller’s own CLI process so it doesn’t linger, but — unlike stop() — must NOT reach any declared-background descendant the process may have spawned (e.g. a task dev instance launched via run_in_background: true); those must survive the replace. See docs/specs/SPEC_BACKGROUND_TASK_TEARDOWN_SURVIVAL_2026_08_20.md.