Struct PersistentInner 

struct PersistentInner {

    proc_status: String,
    proc_exit_code: i32,
    status_version: i32,
    session_id: Option<String>,
    resume_poisoned: Option<String>,
    fork_copy: Option<Relocated>,
    fork_next: bool,
    restart_when_idle: bool,
    restart_pending: bool,
    spawn_runtime: Option<SpawnRuntime>,
    effective_runtime: Option<EffectiveRuntime>,
    settings_readback: bool,
    context_usage: Option<AgentContextUsage>,
    stop_pending: bool,
    config_restart_generation: Option<u64>,
    restart_spawn_for: Option<u64>,
    resume: ResumeState,
    spawning_in_progress: bool,
    pending_send_messages: VecDeque<QueuedMessage>,
    deferred_deliveries: VecDeque<String>,
    deferred_watchdog_armed: bool,
    drain_claim: bool,
    next_message_seq: u64,
    drain_send_in_flight: bool,
    current_pid: Option<u32>,
    stdin_tx: Option<Sender<String>>,
    kill_tx: Option<Sender<KillRequest>>,
    shutdown_generation: Option<u64>,
    stop_exit: Option<(u64, bool)>,
    spawn_generation: u64,
    leftover_resume_candidate: Option<String>,
    pending_questions: HashMap<String, (String, Value)>,
    pending_permissions: HashMap<String, (String, String, Value)>,
    agent_lease: Option<Arc<HeldAgentLease>>,
}

Inner state protected by mutex.

Fields

proc_status: String

proc_exit_code: i32

status_version: i32

session_id: Option<String>

resume_poisoned: Option<String>

A --resume id the CLI has confirmed (via stderr) it can’t find under the current config dir. The stdout reader echoes back whatever --resume value it was given as its first line REGARDLESS of whether resume actually succeeds, racing the stderr reader’s own clear of session_id — this stops that race from re-adopting a known-dead id. Never reset back to None: a genuinely fresh session id (a new CLI- generated UUID) will never equal this one, so a stale poison value is permanently inert rather than something that needs clearing.

fork_copy: Option<Relocated>

The copy the resume gate relocated for the next spawn to --resume … --fork-session from (SPEC_RESUME_GATE_AND_SAME_IDENTITY_CONTINUATION_2026_09_25.md §4.3). Taken by that spawn; removed if it ends up resuming nothing.

fork_next: bool

The next spawn resumes with --fork-session (a relocation, or a session that grew outside AgentMux, spec §4.4). Taken by that spawn.

restart_when_idle: bool

Set when a forced controller resync (a /model, /effort or /permission-mode change — see frontend/.../runtime-apply.ts) lands while a turn is in flight. The restart is DEFERRED to the end of that turn instead of killing the process mid-turn.

Killing it was the old behaviour and it silently destroyed the user’s in-flight message: stop_process records a StopRequested, which the resume state machine treats as an explicit user stop and therefore suppresses the retry; remove_controller_entry_only then discards the controller (and its queue) outright; and the replacement controller “spawns on first message”, so nothing resumed. The message had already been written to the dead process’s stdin, so no queue entry survived to replay. Net effect: the turn vanished with no response and no error — the pane simply went quiet. Diagnosed live on AgentX, 2026-08-28.

Deferring is correct rather than merely safer: model/effort flags are baked in at spawn, so they could not have applied to the running turn anyway. Waiting until it ends costs nothing and loses nothing.

restart_pending: bool

A deferred restart has COMMITTED — stop_process has been called and the process is on its way out, but stdin_tx is still live because stop_process only sends on kill_tx and leaves the writer channel alone until the process actually exits.

Without this, decide_send_action would see a live stdin_tx and take DeliverDirect for any message arriving in that window — writing it into a process about to receive EOF, acknowledging it, and losing it (codex P1 on PR #2858). The window is not hypothetical: the restart fires exactly at turn end, which is precisely when the frontend flushes its queued follow-ups.

Set under the same lock that consumes restart_when_idle, cleared when the replacement process spawns.

spawn_runtime: Option<SpawnRuntime>

The runtime flags (model, effort, permission) in the argv of the process that is running now; None when no process is. What the menu compares its selection against: the menu shows what was asked for, this is what the process was given. Set at spawn, cleared when the status leaves “running” (set_status).

effective_runtime: Option<EffectiveRuntime>

What the running CLI itself reported using (get_settings’s applied): the alias resolved to a concrete model, and the effort in force. None until it answers, for a CLI that cannot, and when no process runs.

settings_readback: bool

Whether the running process takes control requests, so it is asked get_settings and get_context_usage (set at spawn: a control-protocol launch).

context_usage: Option<AgentContextUsage>

The last get_context_usage answer published for the current process, so an unchanged answer at each turn boundary isn’t re-broadcast. Cleared at spawn.

stop_pending: bool

A kill has been requested for the current process (request_stop_on). stdin_tx stays live until it actually exits. Writing an automated message then would put it into the dying process and lose it (codex P2 on #3562 for stop, and the same window on the Stop button). try_write_stdin_locked refuses while this is set, so the message stays queued for the next process.

Set under the same lock as the kill request, cleared when the replacement process spawns, exactly like restart_pending.

config_restart_generation: Option<u64>

The spawn generation a deferred runtime-config restart is replacing: set by turn_boundary_locked in the same acquisition that commits the restart, so the old process’s exit handler knows to bring the replacement up itself once the process is gone.

Without it the pane was left with no process until a human typed into it: automated delivery (send_user_message) never spawns, and it waits while restart_pending is set, so jekts queued forever (Korp on narko, 2026-09-29; docs/retro/RETRO_DEFERRED_RESTART_NEVER_RESPAWNS_2026_09_29.md). A restart is also a stop (stop_pending, StopRequested), so neither of those can tell it apart from a user’s Stop; this can. Every stop request except the restart’s own clears it, and nothing but the boundary sets it, so an explicit Stop or a pane close landing anywhere in the window is never undone by a respawn (#3990). It is consumed only in the acquisition that installs the replacement’s freshly started child as the live process (commit_restart_spawn_locked).

restart_spawn_for: Option<u64>

The spawn claimed for a config restart’s replacement, by generation (try_claim_eager_resume_spawn_for). spawn_process re-checks the restart token against it right before starting the child, so a Stop during the (possibly slow) credential gate still cancels it.

resume: ResumeState

This spawn generation’s stale---resume retry decision, plus any held-back terminal error-result line — see persistent_resume::ResumeState’s own doc comment for the full design rationale. Replaces what used to be four separate fields (pending_resume_retry, confirmed_stale_resume_retry, stop_requested_generation, pending_error_result_line) mutated directly by four independently-scheduled tasks racing on this same mutex — that shape caused issue #2368 (and a live-reproduced recurrence, agent “Marks”, 2026-07-30) because no single owner enforced valid transitions between them. Every mutation now goes through persistent_resume::update(), a pure, exhaustively unit tested (state, event) -> (state, effects) function — still called under this same mutex (no new concurrency primitive is introduced), but as ONE call per event instead of several separate field reads/writes that could observe each other mid-transition.

spawning_in_progress: bool

True from the moment a caller commits to calling spawn_process (in send_message or retry_after_resume_failure) until it has delivered every message enqueued for that spawn — see pending_send_messages. Checked and set together with stdin_tx.is_some() under this same lock, in one acquisition — reagentx P1 on PR #2360 (sixth review pass): send_message’s is_running() check and its spawn_process() call used to be two separate operations; a second concurrent send_message call (a genuine second RPC, or a muxbus delivery) landing in the gap between them could ALSO observe “not running” and independently spawn a second child process, orphaning one (leaked, unkillable via stop_process, unregistered from muxbus). Whichever caller sees stdin_tx.is_none() && !spawning_in_progress first becomes the sole spawner for this round; every other caller queues instead of racing its own spawn.

This also fully subsumes the earlier spawn_epoch/ should_skip_own_delivery mechanism (rounds 3-5 of this same PR): that check existed only to catch a delivery landing in the window between a caller’s own spawn_process returning and its own tail-end stdin write — a window that no longer exists, since delivery now happens as part of the same atomic spawn-claim, before the lock is released (see release_spawn_claim_and_drain_queue).

pending_send_messages: VecDeque<QueuedMessage>

Messages queued while spawning_in_progress was true — includes the enqueuing caller’s own message when it became the spawner (see SendAction::BecomeSpawner), so the post-spawn drain uses one uniform delivery path regardless of whether a message triggered the spawn or arrived while someone else’s spawn was already in flight. Drained by release_spawn_claim_and_drain_queue.

deferred_deliveries: VecDeque<String>

Non-human messages (jekt, muxbus, bridges, MCP SendMessage) that arrived while the process could not take a write — a spawn in flight, a committed restart, a requested stop, or another writer owning stdin (deferred_must_wait_locked). Written, in order, by the watchdog once it can. A live process that can take a write gets the message at once, mid-turn or not (SPEC_JEKT_IMMEDIATE_DELIVERY_2026_09_28.md §2.1), so nothing waits here for a turn to end.

Distinct from pending_send_messages, the human path’s queue around a spawn. Entries are the fully encoded stream-json stdin lines, ready to write. Read and written only under this mutex, together with the must-wait state it depends on, so a message is never enqueued just as a drain observes the queue empty and exits.

deferred_watchdog_armed: bool

Whether a deferred-delivery watchdog task is running for this controller — see ensure_deferred_watchdog. Set and cleared only under this lock, and cleared only by the watchdog itself when it finds the queue empty, so an enqueue can never observe “armed” from a watchdog that has already decided to exit.

drain_claim: bool

Exclusive claim held by a stale-resume retry batch flush (issue #2367; spec §4 option 2 of SPEC_PERSISTENT_SPAWN_GENERATION_AND_MESSAGE_IDENTITY_2026_08_09). Taken by decide_retry_batch_action in the SAME lock acquisition in which it decides a live, newer-generation process should receive the batch; while held, decide_send_action’s DeliverDirect branch routes to Queued instead (exactly how it already treats spawning_in_progress), making the queue the single ordering authority — a concurrent send can no longer try_send ahead of earlier-accepted retry messages. Released by the flush task (drain_queue_with_claim) once the queue runs dry or the process dies. Deliberately a NEW flag, not a reuse of spawning_in_progress — round-14’s starvation analysis stands: that flag is never pre-asserted while no spawn is in progress.

next_message_seq: u64

Monotonic counter behind QueuedMessage::seq (issue #2365) — pre-incremented at every fresh enqueue, so real seqs start at 1 and are never reused for this controller’s lifetime. Redelivery paths (a stale-resume retry batch) preserve a message’s original seq instead of drawing a new one — that preservation is what makes “already queued?” an identity check rather than a content comparison.

drain_send_in_flight: bool

True while the drain (drain_queue_after_successful_spawn) is between successfully sending a message on the live stdin channel and finishing its OWN follow-up append of that message into the stale---resume retry batch held in resume (ResumeEvent::MessageAppendedToRetryBatch) — see the drain’s own comment at that append for why it exists (#2360): the Sender::send().await and that follow-up append are two separate lock acquisitions with an unavoidable gap between them (a mutex can’t be held across an .await). Without this flag, the process-waiter’s own exit-handling — running concurrently on a DIFFERENT task — could resolve the retry batch (ResumeEvent::ProcessExited) in that exact gap, dispatching a retry that’s missing a message the doomed process’s channel had ALREADY accepted: the message stays marked “accepted” and gets persisted, but is never actually delivered to any process again. The exit-handling waits (briefly, bounded) for this to go false before deciding the retry batch is final — see its own comment at that wait loop.

current_pid: Option<u32>

stdin_tx: Option<Sender<String>>

Channel to send messages to the stdin writer task.

kill_tx: Option<Sender<KillRequest>>

Handle to kill the process.

shutdown_generation: Option<u64>

Set by Controller::shutdown to the spawn generation it is closing, BEFORE it interrupts the turn. The interrupted turn ends in an is_error: true result (error_during_execution) that is the stop we asked for, not a failure — the stdout reader must not classify it, raise a failure banner, or feed it to the stale---resume retry machine (spec §9.4).

stop_exit: Option<(u64, bool)>

Set by the kill arm the moment a requested stop’s process has exited: (spawn generation, killed). killed = it had to be force-killed (graceful deadline passed, or a forced stop). Watched by shutdown, which must not wait on the kill arm’s later cleanup (that can take seconds when a descendant holds stdout open).

spawn_generation: u64

Monotonic counter bumped once per spawn_process call (in the same lock acquisition that reports SpawnedWithResume/SpawnedFresh to resume), uniquely identifying that one spawn attempt for the rest of this controller instance’s lifetime. Note this is NOT the spawn_epoch/should_skip_own_delivery mechanism removed earlier (see spawning_in_progress’s own doc comment) — that existed to dedup a spawn-claim race, a job spawning_in_progress now fully owns. This counter exists for a different, narrower purpose: giving every ResumeEvent (and the per-spawn tokens above, such as config_restart_generation) something stable to compare against.

ALSO bumped (without a spawn) by clear_session_id_for_fresh_spawn to atomically retire every existing generation’s reader tasks alongside a session-id clear — see its doc comment (codex P1 on PR #2500, second round). The resulting numbering gap is deliberate and inert: generations are only ever compared for equality; the invariant is “never reused,” not “no gaps.”

leftover_resume_candidate: Option<String>

A recovery candidate adopted by settle_empty_resume_retry while accepted prompts were still queued behind the eager spawn claim (codex P1 on PR #3551, fifth round). Those prompts were never delivered to the doomed process, so they are not in the retry batch — the eager path’s own drain will find the dead stdin_tx, stall, and hand them to respawn_once_for_leftover_queue, whose contract is otherwise “spawn fresh”. This tells that respawn to --resume the candidate instead, so a recoverable session is not thrown away for a prompt that happened to arrive during the eager attempt. Consumed by that respawn; invalidated by any spawn (spawn_process clears it under the generation bump), since a candidate only means anything for the spawn that immediately follows its adoption.

pending_questions: HashMap<String, (String, Value)>

AskUserQuestion can_use_tool control_requests awaiting a user answer: tool_use_id -> (request_id, questions JSON). Filled by the stdout reader when the CLI sends a can_use_tool control_request for AskUserQuestion; consumed by answer_question to build the matching control_response. Spec: docs/specs/SPEC_AGENT_CONTROL_PROTOCOL_2026_06_15.md.

pending_permissions: HashMap<String, (String, String, Value)>

Ordinary (non-AskUserQuestion) tool-permission can_use_tool control_requests awaiting a user decision: tool_use_id -> (request_id, tool_name, input JSON). Filled by park_tool_permission_request when should_route_to_decision_panel says so; consumed by decide_tool_permission to build the matching control_response. Structurally the sibling of pending_questions above — same in-memory-only, per-controller-instance lifetime — but currently always empty in production: should_route_to_decision_panel is hardcoded false pending the Phase 2 policy decision (SPEC_DECISION_PROMPT_2026_04_24.md, SPEC_AGENT_CONTROL_PROTOCOL_2026_06_15.md §5).

agent_lease: Option<Arc<HeldAgentLease>>

The single-live-instance lease the current CLI process holds (backend::agent_admission, SPEC_AGENT_SINGLE_LIVE_INSTANCE_2026_09_24 Phase 1). Set by spawn_process once the child is running; cleared — which releases it — by the process-exit arms for the current generation. An Arc so the pre-turn fence can verify it without holding this lock across file I/O.

Implementations

impl PersistentInner

fn take_next_message_seq(&mut self) -> u64

Draws the next message seq (issue #2365) — pre-incremented, so real seqs start at 1 and 0 can serve as “never a valid seq” in tests. Must be called under the same lock acquisition as the enqueue it identifies.

fn apply_resume_event(&mut self, event: ResumeEvent) -> Vec<ResumeEffect>

Applies one persistent_resume::ResumeEvent to self.resume, storing the resulting state back and returning the effects for the caller to execute. The one and only place self.resume is ever mutated — see persistent_resume’s module doc comment for why routing every transition through this single pure function (instead of several fields each caller used to read/write directly) closes the exact race class issue #2368 kept resurfacing.

fn poison_resume(&mut self, bad_sid: &str, generation: u64)

Records bad_sid as confirmed-unreachable (the CLI reported “No conversation found” for it) and clears it from session_id if it’s currently held there. Pairs with try_capture_session_id below — whichever of the stderr/stdout reader tasks runs first, the poisoned id never survives as the live session_id. generation is the spawn attempt this poison applies to — see persistent_resume::ResumeEvent’s own doc comment for why every event carries one.

fn try_capture_session_id( &mut self, sid: &str, generation: u64, is_confirmed_success: bool, ) -> (bool, Vec<ResumeEffect>)

Attempts to adopt sid as the live session id. Returns false (does not adopt) if a session id is already held, or if sid is the confirmed-poisoned id from a prior poison_resume call — the CLI echoes back whatever --resume it was given as its first stdout line even when that resume goes on to fail, so without this check a losing race would silently re-adopt a known-dead id right after poison_resume cleared it. A genuinely different (fresh) sid is unaffected and still captured normally. generation is the spawn attempt this capture applies to.

codex P1 on PR #2371: resolves the tentative/confirmed retry tracking whenever sid is confirmed genuine — NOT only on the adopted (session_id was previously None) branch. A --resume <sid> spawn ALWAYS has session_id already Some BEFORE the process even starts (that’s what makes --resume get attached at all — see spawn_process), so on the common resume-SUCCEEDED case the CLI’s echoed sid matches what’s already held and this always used to return false immediately, WITHOUT ever resolving tracking. Since persistent mode never exits between turns, that would leave a generation’s resume state live for the rest of this potentially long-lived process’s life, wrongly holding back every LATER, completely unrelated is_error:true result as if it might still need to be dropped for a stale-resume retry.

reagentx P0 on PR #2371: is_confirmed_success (true only for a terminal result with is_error:false) is threaded through to persistent_resume::update, which is what ACTUALLY decides whether this capture is unambiguous enough to resolve tracking — see ResumeEvent::SessionCaptured’s own doc comment for why a same-sid echo alone (e.g. an early “system”/init frame) is NOT proof of genuine progress: the CLI echoes the attempted sid regardless of whether the resume goes on to fail.

reagentx P0 on PR #2373: returns the effects apply_resume_event produced, not just the adopted bool — resolving tracking here can legitimately emit FlushErrorLine (an EARLIER turn on this same still-alive generation held an error line back before this LATER capture resolved tracking — see SessionCaptured’s own handling in persistent_resume::update). Discarding that silently lost the held-back line instead of flushing it, reproducing the exact #2368 bug class this PR exists to fix. The caller (the stdout reader) is responsible for actually executing it, same as every other ResumeEffect this module produces.

fn holds_current_session(&self, sid: &str, generation: u64) -> bool

The process of generation is still the current one, and still holds sid as a session not known to be dead. A superseded generation’s draining stdout must not settle anything on the replacement’s behalf (codex P1 on #3907).