Skip to main content

Module m0024_native_memory_backfill_from_fs

Module m0024_native_memory_backfill_from_fs 

Source
Expand description

Finish the job m0023 was meant to do, and repair the label it left behind — SPEC_MEMORY_VERSION_CONTROL_AND_ARMORY_AUDIT_2026_08_19.md §9 (rollout backfill) and §4.5 (drift detection).

§What went wrong

§9 asks for “a one-time backfill [of] a single source: 'agent_inferred' version per existing (agent_id, filename) row … so history views aren’t empty for pre-existing memory on upgrade”. m0023 implements that literally, iterating db_agent_native_memory — the mirror table. The spec’s own wording (“per existing … row”) is what led there, and it is wrong in practice for one reason: that mirror is populated lazily, only once something reads or exports an agent’s memory. On a typical install it is EMPTY, so m0023 iterates nothing and backfills nothing, in 0 ms.

The drift sweep (backend::native_memory_drift) does not use the mirror — it reads the real filesystem, which is authoritative here (SPEC_NATIVE_MEMORY_DURABLE_SYNC_2026_08_07.md). Starting moments after migrations finish, it finds those same files, sees no recorded version, and records each as external_fs_write — rendered in Memory as “Detected outside AgentMux — provenance unknown”.

Net effect on every install that upgraded with an unpopulated mirror: ordinary pre-existing memory files are permanently tagged with a security-flavored warning, and the benign backfill the spec asked for never happened. Observed on a real install: m0023 applied at …285933 (0 ms, 0 rows, mirror empty), 21 external_fs_write rows written across 6 agents and 16 files 61 ms later at …285994.

This is NOT a race — bootstrap runs migrations to completion before the drift tasks spawn. Ordering was never the problem; reading the wrong source was. That is why the fix below is a migration and not a change to startup sequencing.

§What this does

  1. (Moved out of this migration by SPEC_MEMORY_FOLLOWS_THE_AGENT phase M1 — see up.) Backfill from the filesystem, the same source the detector trusts, via the same list_all_memory_targets enumeration — so the two agree by construction instead of by coincidence. Idempotent and ordering-safe for exactly the reason m0023 documents: only a pair with NO version yet is touched, so a pair written normally since upgrade never gets an out-of-chronology row appended.
  2. Repair the mislabeled rows already written by the sweep, under a deliberately narrow predicate — see agent_native_memory_version_relabel_rollout_first_sight, which documents each clause. It cannot touch a genuine external write.

§Why this is Channel-scoped, and which store enumerates

The two stores are not interchangeable, and picking the wrong one here would have reproduced the exact bug this migration exists to fix (caught in review before merge, not by me):

  • Versions (db_agent_native_memory_versions) live in the SHARED store, so every read/insert/relabel below uses shared_store_path.
  • Enumeration does not. list_all_memory_targets calls agent_def_list(), which prepares against db_agents, and memory_dir_for_agent_by_id, which reads db_agent_content — both per-channel tables created by run_object_schema, absent from Store::open_shared by design (see its doc comment). Handing it the shared store makes agent_def_list() fail with a SQL error that list_all_memory_targets swallows (if let Ok(agents)), so that whole enumeration path silently yields nothing and only currently active registry agents get backfilled — every persisted-but-idle agent missed, which is the same “read the wrong source, get an empty list, look successful” failure as m0023.

So the migration is Channel-scoped: it runs once per channel, with that channel’s own store doing the enumeration — matching what the drift sweep itself gets in production (state.mstore is the channel store) — while still reading and writing versions in the shared store. The repair half is idempotent, so running once per channel is safe.

Order matters: repair runs FIRST. Relabeling turns a row into the agent_inferred version the pair should have had, which then makes step 2’s “has no version yet” test correctly skip that pair. Backfilling first would leave the pair versioned but still mislabeled, and the repair’s parent_version_id IS NULL clause would no longer match.

Structs§

M0024NativeMemoryBackfillFromFs