--- title: @memhtml/head --- `@memhtml/head`: the corpus as a value, one version per commit held in persistent `effect` structures. A version is loaded from git, advanced by the paths two commits differ on, and overlaid with a session's pending ops. It is searched by reciprocal-rank fusion over a lexical arm and an optional vector arm, with recency breaking exact ties, and the head gate compares two versions. The design is `docs/design.md` §16. `HeadVersion` is one version: the read-only view interface `@memhtml/contracts` defines, answered from the version's `Indexes`, plus its commit `sha` and its `parent` version (a pointer, so a caller can walk back). `skippedFiles` lists the candidate files left out because they failed to parse or their blob was missing from the batch read, each a `SkippedFile` with its path and reason, recorded rather than thrown, and `skipped` is their count. `isCandidatePath` decides which tree paths are parsed: a path ending in `.html`, not under a top-level `.memhtml/`, whose file name is not `index.html`. `loadHead` builds the version at a commit from a full tree read: one `ls-tree` and one batched `cat-file` read through `@memhtml/store`'s git service, folding `insertRecord` over the parsed records in tree order. `advanceHead` builds the version at a later commit from its parent. It lists the paths the two trees differ on with one `git diff-tree -r -z --name-only --no-renames`, so a rename contributes both of its paths. It removes the changed candidates absent at the new commit and re-parses the rest unless the blob sha equals the parent record's, and each unchanged path keeps the parent's record object. A changed file that fails to parse leaves the version and joins `skippedFiles`, and a target equal to the parent's `sha` returns the parent itself. `versionFromRecords` builds a version from records already parsed, with no parent and `skipped` 0, by the same `insertRecord` fold. `recordFrom` turns one file into the flat record type `@memhtml/contracts` defines, and the versions and overlays in this package parse through it. It runs the `@memhtml/html` parser and fails with the parser's own error. To the parsed fields it adds the git blob sha of the text it is given, the article's content hash, the claim's frame key, and the archive flag from `isArchivedPath`, which is true for a path under `archive/`. `blobShaOf` computes the blob sha locally, sha1 over `blob \0` followed by the UTF-8 bytes, because an overlay holds files git has not stored yet. It counts UTF-8 bytes rather than string length, so multibyte content agrees with git. `Indexes` is the record map and the indexes derived from it, as one value of persistent `effect` maps and sets, so a change to one record builds a new value that shares the untouched structure. Its fields are `records` by path, `byContentHash` (content hash to an active path holding it), `byFrameKey` (frame key to the active paths whose claim shares it), `inbound` (href, as authored, to the paths linking to it), `byEntity` (entity to the paths naming it), and `lexical`. `emptyIndexes` is the empty value, and `insertRecord` and `removeRecord` are its two writers. `insertRecord` replaces any record at the same path, and an archived record enters only `records`, `inbound` and `byEntity`. A content hash goes to the first active record inserted while no path holds it; removing another record that shares it leaves it held, and removing the holder leaves it unmapped even while a twin stays active. `viewOf` puts the read surface over one `Indexes` value and a sha, with `byFrameKey`, `inbound` and `byEntity` returning paths sorted ascending. `indexesOf` returns the indexes behind a view: it reuses them for a view built here and rebuilds them by `insertRecord` over its records otherwise, which is how `searchHead` and `withOverlay` accept a caller's own view. `LexicalIndex` is the lexical arm's persistent index over active records: per-token postings with each path's term frequency, and per-path token counts with their total, so BM25 reads term frequency and average length at query time without re-tokenizing a body. A record is indexed by its title, claim and body text. `tokenize` lowercases and splits on runs of characters other than Unicode letters, numbers and underscore. It drops empty tokens and the 57 `STOP_WORDS`, a closed list of English function words kept short because a longer one starts dropping content words a memory corpus is searched by. `withOverlay` is a session's view: the base view with an ordered list of overlay ops (the op type `@memhtml/contracts` defines) applied through `insertRecord` and `removeRecord`. It keeps the base's `sha`, because an overlay has no commit of its own, and fails on the first op that does not parse or resolve. `put` parses a file and inserts it at its path. `archive` removes the old path and inserts the given html, parsed at the destination path, as an archived record. `link` and `unlink` add or cut an edge in the file's head, and `label` and `unlabel` a `memhtml-entity` meta, then re-parse the file; a `link` or `label` naming an edge or entity the file already carries changes nothing. All four fail on a path the view does not hold, `link` and `unlink` also on a rel outside the edge vocabulary, `label` on a malformed entity, and `unlink` and `unlabel` on an edge or entity the file does not carry. `move` re-parses the record's current html, including earlier ops' head edits, at its new path, and fails unless the old path is held and not archived. `searchHead` ranks the active records of a view and returns `SearchHit` rows. It is synchronous and does not embed, so the caller owns any vectors. A `SearchHeadInput` carries the `query`, a `limit` (default `DEFAULT_LIMIT`, 10), an optional `now`, an optional `admit` predicate, and optionally `vectors`, a `HeadVectors` whose `get` looks a vector up by content hash, with a `queryVector` in the same space. `admit` narrows the records each arm ranks before its cap, while BM25's statistics stay those of all active records. The lexical arm is BM25 (k1 of 1.2, b of 0.75) over the lexical index, counting each distinct query token once, and keeps its best `LEXICAL_ARM_LIMIT` (40). The vector arm runs only when both `vectors` and `queryVector` are set: it scores each admitted active record that has a vector by cosine against the query vector and keeps its best `VECTOR_ARM_LIMIT` (40). Within an arm, records with equal scores share the best rank of their run (1, 1, 3). The arms are fused by reciprocal-rank fusion at k = 60, with `LEXICAL_WEIGHT` and `VECTOR_WEIGHT` both 1.0, and without a vector arm the fold is the lexical arm alone. Recency only breaks exact ties. Hits are ordered by fused score, then newest first by the record's event time, or its update time when it has none (compared as strings, and clamped to `now` when the input sets it), then by path ascending. A score difference is not overturned and no record is added, so a query no arm matches returns nothing. Each `SearchHit` carries its `path`, fused `score`, `claim`, and `arms`, the `SearchArm` values behind its place: `lexical` and `vector` for the arms that ranked it, and `recency` when its fused score equals another candidate's, which is when recency or path decided its position. `fillVectors` is the one step in the package that calls an embedder, which keeps `searchHead` free of one. `vectorTextOf` is the text a record embeds: its article text, trimmed and cut to the first `VECTOR_TEXT_MAX_CHARS` (1,800) UTF-16 code units, the same number as `@memhtml/index`'s chunk ceiling. The title is left out because it lives in the file's head, outside what the content hash digests, so a vector that read it could go stale under a key that does not change. A `VectorCache` holds the vectors of one `VectorSpace` (a `modelId` and a `dimension`) by content hash, so an unchanged article keeps its vector across versions, moves and head-only edits; `emptyVectorCache` starts one. `fillVectors` embeds, through a `DocumentEmbedder`, the active records whose content hash the cache lacks. `DocumentEmbedder` is the half of `@memhtml/llm`'s embeddings service a fill needs, typed structurally so this package does not import `@memhtml/llm`. A fill makes at most one `embed` call, with one text per distinct content hash, and skips archived records and records whose trimmed article text is empty. It returns a `FillResult`: the cache with the new vectors added beside the ones it held (vectors for hashes the view no longer holds stay), the `FillCounts`, and `remaining`, the distinct texts a `limit` held back for a later fill. An embedder that answers with the wrong number of vectors or a vector of the wrong width ends the fill as a defect instead of storing anything. `gateHeads` is the head gate: it asks whether a landed version still finds what its base found, over two versions of the same corpus rather than a fixture. `probesFor` takes the records active in both versions as `HeadGateProbe` rows (a path and a query), so a record the landing adds or archives is not probed. It orders them by the first eight hex digits of sha1 over `:`, with path breaking ties, and keeps the first `count`, so the sample is a fixed function of the seed and the paths. `probeQueryFor` gives each probe its query from the landed version's record: the title, else the first sentence of the claim, else the claim. `scoreProbes` runs `searchHead` at one version for each probe with a limit of `k` and no vectors, so the gate needs no embedder. `scoreProbes` returns the ranks, the MRR (rounded to four places, 0 with no probes) and two inversion kinds: `fell-out` when the target is not in the top `k`, and `superseded-outranks` when a record named by one of the target's `supersedes` links ranks above the target in the top `k`, or appears there while the target does not. `gateHeads` scores the same probes at both versions. It passes when the landed MRR reaches the `floor` and the landed version has no more inversions than the base. The `floor` is the base MRR minus the tolerance, clamped at zero and rounded to four places; the tolerance allows for a landing that archives a near-duplicate and so shifts BM25's document frequencies slightly. `HeadGateOptions` overrides `probes`, `k`, `tolerance` and `seed`, which default to `HEAD_GATE_PROBES` (200), `HEAD_GATE_K` (10), `HEAD_GATE_TOLERANCE` (0.02) and `HEAD_GATE_SEED` (1). A `HeadGateReport` carries both MRRs, the `floor`, the landed inversions beside the base's count, and one `HeadGateResult` per probe. `describeHeadGateFailure` renders a report as one line: the probe count, then the MRR shortfall and the inversion growth (with up to five inversions) when they occurred, or `passed`. `HeadGateFailed` wraps a report as a value tagged `HeadGateFailed` whose `reason` is that line, which the CLI maps to `ERR_DISCRIMINATION_FAILED`. ## Modules The package publishes one import path, `@memhtml/head`, whose `index.ts` re-exports a chosen set of names from the other eight modules in the table below, so the reference on this page is the whole exported surface. Names a module defines that `index.ts` does not re-export are left off the page, which is why the five the table names as not exported appear unlinked where a signature or doc comment mentions them. | Module | What it holds | | ------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `gate.ts` | The defaults (`HEAD_GATE_PROBES`, `HEAD_GATE_K`, `HEAD_GATE_TOLERANCE`, `HEAD_GATE_SEED`), the shapes `HeadGateOptions`, `HeadGateProbe`, `HeadGateInversion`, `HeadGateResult` and `HeadGateReport`, `probeQueryFor`, `probesFor`, `scoreProbes`, `gateHeads`, `describeHeadGateFailure` and `HeadGateFailed`, plus `Scored`, the result type `scoreProbes` returns, which is not exported. | | `indexes.ts` | `Indexes`, `emptyIndexes`, `insertRecord`, `removeRecord`, `viewOf` and `indexesOf`, plus `IndexedView`, the view type `viewOf` returns and `HeadVersion` extends, which is not exported from `@memhtml/head`. | | `lexical.ts` | `LexicalIndex`, `STOP_WORDS` and `tokenize`, plus `addToLexical` and `removeFromLexical`, the lexical writers `insertRecord` and `removeRecord` call, which are not exported from `@memhtml/head`. | | `overlay.ts` | `withOverlay`. | | `record.ts` | `recordFrom`, `blobShaOf` and `isArchivedPath`. | | `search.ts` | The shapes `SearchArm`, `SearchHit`, `HeadVectors` and `SearchHeadInput`, the constants `LEXICAL_WEIGHT`, `VECTOR_WEIGHT`, `LEXICAL_ARM_LIMIT`, `VECTOR_ARM_LIMIT` and `DEFAULT_LIMIT`, and `searchHead`, plus `armRanks`, the shared-rank step both arms use, which is not exported. | | `vectors.ts` | `VECTOR_TEXT_MAX_CHARS`, `vectorTextOf`, the shapes `VectorSpace`, `VectorCache`, `DocumentEmbedder`, `FillCounts` and `FillResult`, `emptyVectorCache` and `fillVectors`. | | `version.ts` | `HeadVersion`, `SkippedFile`, `isCandidatePath`, `loadHead`, `versionFromRecords` and `advanceHead`. | | `index.ts` | The re-exports above, type-only for the interfaces and type aliases. | ## Classes ### HeadGateFailed A failed head gate, as a tagged error the CLI maps to `ERR_DISCRIMINATION_FAILED`, the code every refused landing carries. The whole report rides along, because a refusal an operator cannot read is a refusal they override. #### Constructors ##### Constructor ```ts new HeadGateFailed(report): HeadGateFailed; ``` ###### Parameters ###### report [`HeadGateReport`](/api/head/#headgatereport) ###### Returns [`HeadGateFailed`](/api/head/#headgatefailed) #### Properties ##### \_tag ```ts readonly _tag: "HeadGateFailed" = "HeadGateFailed"; ``` ##### report ```ts readonly report: HeadGateReport; ``` #### Accessors ##### reason ###### Get Signature ```ts get reason(): string; ``` ###### Returns `string` ## Interfaces ### DocumentEmbedder The half of `@memhtml/llm`'s `EmbeddingsShape` a fill needs, structurally, so this package never imports it. #### Type Parameters ##### E `E` #### Properties ##### embed ```ts readonly embed: (texts) => Effect[], E>; ``` ###### Parameters ###### texts readonly `string`[] ###### Returns `Effect`\[], `E`\> *** ### FillCounts What one fill did, counted over the view's active records. #### Properties ##### embedded ```ts readonly embedded: number; ``` Distinct texts sent to `embed`; two active records sharing a content hash cost one. ##### empty ```ts readonly empty: number; ``` Active records with no text to embed (an empty article), which the arm never ranks. ##### records ```ts readonly records: number; ``` Active records in the view. ##### reused ```ts readonly reused: number; ``` Active records whose content hash already had a vector. *** ### FillResult #### Properties ##### cache ```ts readonly cache: VectorCache; ``` ##### counts ```ts readonly counts: FillCounts; ``` ##### remaining ```ts readonly remaining: number; ``` Distinct texts still missing after this fill: 0 unless `limit` held some back, which the next fill picks up. *** ### HeadGateOptions `@memhtml/head`: the corpus as a value. One immutable version per commit, held as persistent `effect` structures, advanced by delta, read through `HeadView`, searched by RRF over a lexical and (given vectors) a vector arm, with recency breaking exact ties. See `docs/design.md` §16. #### Properties ##### k? ```ts readonly k?: number; ``` The top-`k` a target must appear in. ##### probes? ```ts readonly probes?: number; ``` How many probes to sample. Fewer when the comparable set is smaller. ##### seed? ```ts readonly seed?: number; ``` ##### tolerance? ```ts readonly tolerance?: number; ``` How far below the base MRR the landed MRR may sit and still pass. *** ### HeadGateProbe `@memhtml/head`: the corpus as a value. One immutable version per commit, held as persistent `effect` structures, advanced by delta, read through `HeadView`, searched by RRF over a lexical and (given vectors) a vector arm, with recency breaking exact ties. See `docs/design.md` §16. #### Extended by - [`HeadGateResult`](/api/head/#headgateresult) #### Properties ##### path ```ts readonly path: string; ``` ##### query ```ts readonly query: string; ``` *** ### HeadGateReport `@memhtml/head`: the corpus as a value. One immutable version per commit, held as persistent `effect` structures, advanced by delta, read through `HeadView`, searched by RRF over a lexical and (given vectors) a vector arm, with recency breaking exact ties. See `docs/design.md` §16. #### Properties ##### added ```ts readonly added: number; ``` Active records at the landed version that were not active at the base. ##### base ```ts readonly base: string | null; ``` ##### floor ```ts readonly floor: number; ``` `mrrBefore - tolerance`, clamped at zero. The landed MRR must reach it. ##### inversions ```ts readonly inversions: readonly HeadGateInversion[]; ``` ##### inversionsBefore ```ts readonly inversionsBefore: number; ``` ##### k ```ts readonly k: number; ``` ##### landed ```ts readonly landed: string | null; ``` ##### mode ```ts readonly mode: "head"; ``` ##### mrr ```ts readonly mrr: number; ``` Mean reciprocal rank of the probes at the landed version, four decimals; `0` with no probes. ##### mrrBefore ```ts readonly mrrBefore: number; ``` The same probes at the base version. ##### passed ```ts readonly passed: boolean; ``` ##### probes ```ts readonly probes: number; ``` ##### removed ```ts readonly removed: number; ``` Active records at the base that are not active at the landed version. ##### results ```ts readonly results: readonly HeadGateResult[]; ``` ##### seed ```ts readonly seed: number; ``` ##### tolerance ```ts readonly tolerance: number; ``` *** ### HeadGateResult `@memhtml/head`: the corpus as a value. One immutable version per commit, held as persistent `effect` structures, advanced by delta, read through `HeadView`, searched by RRF over a lexical and (given vectors) a vector arm, with recency breaking exact ties. See `docs/design.md` §16. #### Extends - [`HeadGateProbe`](/api/head/#headgateprobe) #### Properties ##### path ```ts readonly path: string; ``` ###### Inherited from [`HeadGateProbe`](/api/head/#headgateprobe).[`path`](/api/head/#headgateprobe) ##### query ```ts readonly query: string; ``` ###### Inherited from [`HeadGateProbe`](/api/head/#headgateprobe).[`query`](/api/head/#headgateprobe) ##### rankAfter ```ts readonly rankAfter: number | null; ``` ##### rankBefore ```ts readonly rankBefore: number | null; ``` *** ### HeadVectors Vectors by content hash, the key `vectors.ts` gives each one. A `ReadonlyMap` fits; so does any lookup a long-lived head process holds, which is why the input is this narrow. #### Methods ##### get() ```ts get(contentHash): ArrayLike | undefined; ``` ###### Parameters ###### contentHash `string` ###### Returns `ArrayLike`\<`number`\> \| `undefined` *** ### HeadVersion One immutable version of the corpus, built from one commit. `loadHead` builds the first version from a full tree read; `advanceHead` builds every later one from its parent plus the paths the two commits differ on, sharing every untouched record and index node with the parent. `parent` is kept so a caller can walk back; it is a pointer, not a copy, so a chain of versions costs the deltas and nothing more. #### Extends - `IndexedView` #### Properties ##### \[INDEXES\] ```ts readonly [INDEXES]: Indexes; ``` ###### Inherited from `IndexedView.[INDEXES]` ##### lexical ```ts readonly lexical: LexicalIndex; ``` ###### Inherited from `IndexedView.lexical` ##### parent ```ts readonly parent: HeadVersion | null; ``` ##### sha ```ts readonly sha: string; ``` The commit this view was built from; null for a view with no git ancestry (tests, overlays on nothing). ###### Overrides `IndexedView.sha` ##### size ```ts readonly size: number; ``` ###### Inherited from `IndexedView.size` ##### skipped ```ts readonly skipped: number; ``` Files that were candidates and failed `parseMemory`. Skipped, counted, never thrown. ##### skippedFiles ```ts readonly skippedFiles: readonly SkippedFile[]; ``` #### Methods ##### byContentHash() ```ts byContentHash(hash): string | undefined; ``` Active (non-archived) record holding this content hash, if any. ###### Parameters ###### hash `string` ###### Returns `string` \| `undefined` ###### Inherited from `IndexedView.byContentHash` ##### byEntity() ```ts byEntity(entity): readonly string[]; ``` ###### Parameters ###### entity `string` ###### Returns readonly `string`[] ###### Inherited from `IndexedView.byEntity` ##### byFrameKey() ```ts byFrameKey(key): readonly string[]; ``` Active records whose claim shares this frame key. ###### Parameters ###### key `string` ###### Returns readonly `string`[] ###### Inherited from `IndexedView.byFrameKey` ##### get() ```ts get(path): MemoryRecord | undefined; ``` ###### Parameters ###### path `string` ###### Returns `MemoryRecord` \| `undefined` ###### Inherited from `IndexedView.get` ##### inbound() ```ts inbound(href): readonly string[]; ``` Paths whose links point at this href (root-relative form). ###### Parameters ###### href `string` ###### Returns readonly `string`[] ###### Inherited from `IndexedView.inbound` ##### paths() ```ts paths(): Iterable; ``` ###### Returns `Iterable`\<`string`\> ###### Inherited from `IndexedView.paths` ##### records() ```ts records(): Iterable; ``` ###### Returns `Iterable`\<`MemoryRecord`\> ###### Inherited from `IndexedView.records` *** ### Indexes The persistent record map and every index derived from it, as one value. Each field is an `effect` `HashMap` or `HashSet`, so a change to one record builds a new `Indexes` that shares every untouched node with the old one. `insertRecord` and `removeRecord` are the only two writers; every version and every overlay is a fold of those two over a parent. #### Properties ##### byContentHash ```ts readonly byContentHash: HashMap; ``` content hash -> the ACTIVE path holding it. Archived records never enter. ##### byEntity ```ts readonly byEntity: HashMap>; ``` entity (`type:name`, as authored) -> paths naming it. All records. ##### byFrameKey ```ts readonly byFrameKey: HashMap>; ``` frame key -> ACTIVE paths whose claim shares it. ##### inbound ```ts readonly inbound: HashMap>; ``` href (root-relative, as authored) -> paths whose links point at it. All records. ##### lexical ```ts readonly lexical: LexicalIndex; ``` Active records only, see `lexical.ts`. ##### records ```ts readonly records: HashMap; ``` *** ### LexicalIndex The lexical index: token to the paths that carry it, with each path's term frequency, plus the per-document lengths BM25 normalizes by. Every field is a persistent `effect` structure, so a version built from its parent by removeFromLexical and addToLexical shares every bucket the change did not touch. The bucket for one token is `HashMap` rather than a bare path set because BM25 needs the term frequency at query time and re-tokenizing a candidate per query would cost the whole body. #### Properties ##### lengths ```ts readonly lengths: HashMap; ``` path -> token count after stop-word removal. Its size is the document count. ##### postings ```ts readonly postings: HashMap>; ``` token -> path -> term frequency. Active records only. ##### totalLength ```ts readonly totalLength: number; ``` Sum of every length, so the average length is O(1) at query time. *** ### SearchHeadInput #### Properties ##### admit? ```ts readonly admit?: (record) => boolean; ``` Which active records the search is over. Applied inside each arm, before the arm's cap, so a scope narrows the candidates every arm ranks rather than the fused list after it (v1's `assembleScope` reaches every arm the same way). BM25's statistics stay those of every active record, as FTS5's do under a v1 scope. Absent admits every active record. ###### Parameters ###### record `MemoryRecord` ###### Returns `boolean` ##### limit? ```ts readonly limit?: number; ``` ##### now? ```ts readonly now?: string; ``` ##### query ```ts readonly query: string; ``` ##### queryVector? ```ts readonly queryVector?: ArrayLike; ``` The query in the same vector space as `vectors`. ##### vectors? ```ts readonly vectors?: HeadVectors; ``` Vectors by content hash. The vector arm runs only when this and `queryVector` are both set. *** ### SearchHit #### Properties ##### arms ```ts readonly arms: readonly SearchArm[]; ``` The arms that ranked the hit, and `recency` when its fused score equals another candidate's, which is when recency (or path after it) decided its place. ##### claim ```ts readonly claim: string; ``` ##### path ```ts readonly path: string; ``` ##### score ```ts readonly score: number; ``` *** ### SkippedFile #### Properties ##### path ```ts readonly path: string; ``` ##### reason ```ts readonly reason: string; ``` *** ### VectorCache Vectors of one space, by content hash. #### Properties ##### space ```ts readonly space: VectorSpace; ``` ##### vectors ```ts readonly vectors: ReadonlyMap>; ``` *** ### VectorSpace One embedder's vector space. Two caches with different spaces hold incomparable vectors. #### Properties ##### dimension ```ts readonly dimension: number; ``` ##### modelId ```ts readonly modelId: string; ``` ## Type Aliases ### HeadGateInversion ```ts type HeadGateInversion = | { kind: "fell-out"; path: string; query: string; } | { by: string; byRank: number; kind: "superseded-outranks"; path: string; query: string; rank: number | null; }; ``` `@memhtml/head`: the corpus as a value. One immutable version per commit, held as persistent `effect` structures, advanced by delta, read through `HeadView`, searched by RRF over a lexical and (given vectors) a vector arm, with recency breaking exact ties. See `docs/design.md` §16. #### Union Members ##### Type Literal ```ts { kind: "fell-out"; path: string; query: string; } ``` *** ##### Type Literal ```ts { by: string; byRank: number; kind: "superseded-outranks"; path: string; query: string; rank: number | null; } ``` ###### by ```ts readonly by: string; ``` The superseded record that ranked above its superseder. ###### byRank ```ts readonly byRank: number; ``` ###### kind ```ts readonly kind: "superseded-outranks"; ``` ###### path ```ts readonly path: string; ``` ###### query ```ts readonly query: string; ``` ###### rank ```ts readonly rank: number | null; ``` The target's rank, `null` when it was not in the top `k`. *** ### SearchArm ```ts type SearchArm = "lexical" | "recency" | "vector"; ``` Retrieval over one view: a lexical arm and, when the caller passes vectors and a query vector, a vector arm, fused with reciprocal rank fusion at `k = 60`, weights 1.0 and 1.0, over active records only, with recency as the tie-break. The lexical arm is BM25 (Robertson/Sparck Jones idf, `k1 = 1.2`, `b = 0.75`) over the title, the claim, and the body, read from the version's persistent lexical index, and keeps the best [LEXICAL\_ARM\_LIMIT](/api/head/#lexical_arm_limit). The vector arm orders the active records that have a vector by cosine against the query vector, brute force, and keeps the best [VECTOR\_ARM\_LIMIT](/api/head/#vector_arm_limit). In both arms records with equal scores share one rank (see armRanks), so two records an arm cannot tell apart get the same fused share. The rule: recency decides only between hits whose fused score is exactly equal, newest first by `eventAt ?? updatedAt` (clamped to `now`), and path ascending decides after that. It never overturns a score difference and never adds a record, so a query no arm matches returns nothing. A byte-identical newer twin therefore outranks the older copy, and a clearly better older match stays above a newer, weaker one. Why not a weighted recency arm. Until 2026-09-28 recency was an RRF arm at weight 0.5 that ranked every lexical and vector candidate newest first. That made it a second full ranking rather than a tie-break, and a curation landing that made the surviving records newer made older targets lose to newer, weaker matches. On a 2026-09-28 rehearsal of a large collapse on a copy of a production store (the head gate's 200 title probes, seed 1, top 10) the weighted arm scored MRR 0.5572 at the base and 0.4384 landed, so the gate refused the landing; this rule scores 0.9449 and 0.9485 without the vector arm and 0.9492 and 0.9522 with it, and on a 2026-09-23 copy of the same store it raised the gate probes from 0.4557 to 0.9975 without the vector arm and from 0.8321 to 0.9905 with it. Smaller weights land near the same numbers (0.05 scored 0.9374 and 0.9410 without the vector arm), but without shared ranks no weight small enough to be a tie-break lets a newer twin win, and letting recency reorder hits within 0.0003 of each other dropped two-arm MRR at the base to 0.6769 (`docs/design.md` §16, "Recency as the tie-break"). The function stays pure: it never embeds. The caller owns the vectors (keyed by content hash, see `vectors.ts`) and the query vector, so a search with neither is the lexical search exactly. ## Variables ### DEFAULT\_LIMIT ```ts const DEFAULT_LIMIT: 10 = 10; ``` *** ### emptyIndexes ```ts const emptyIndexes: Indexes; ``` *** ### HEAD\_GATE\_K ```ts const HEAD_GATE_K: 10 = 10; ``` *** ### HEAD\_GATE\_PROBES ```ts const HEAD_GATE_PROBES: 200 = 200; ``` The head gate: does the version being landed still find what the base version found? The gate is stated over two versions of the same corpus, never over a fixture. Probes are a seeded sample of the records active in BOTH versions, each queried by its own title (or the first sentence of its claim when the title is blank), and `searchHead` is scored at the base and at the landed version with the same probes, so the two numbers differ only by what the landing changed. A record the landing adds is not probed: it has no rank at the base to compare against, and it becomes a probe at the next landing, when it is part of the base. A record the landing archives leaves the probe set for the same reason, and the probe of the record it was folded into is what measures whether the fold helped. Two inversion kinds, both counted at both versions: a probe whose target is not in the top `k` (`fell-out`), and a record the target supersedes (a `supersedes` link from the target) ranking above the target (`superseded-outranks`). The gate passes when the landed MRR is at least the base MRR minus `tolerance` and the inversion count did not grow. The tolerance exists because a landing that archives a near-duplicate legitimately reshuffles BM25's document frequencies by a hair; a drop past it is a landing that made the corpus harder to search. Everything here is deterministic and credential-free: the sample is fixed by `seed`, both arms of `searchHead` are deterministic, and no embedder is involved. *** ### HEAD\_GATE\_SEED ```ts const HEAD_GATE_SEED: 1 = 1; ``` *** ### HEAD\_GATE\_TOLERANCE ```ts const HEAD_GATE_TOLERANCE: 0.02 = 0.02; ``` *** ### LEXICAL\_ARM\_LIMIT ```ts const LEXICAL_ARM_LIMIT: 40 = DEFAULT_ARM_LIMIT; ``` How many records the lexical arm ranks: v1's `DEFAULT_ARM_LIMIT`, for the vector arm's reason. BM25 scores every record that shares one token with the query, so uncapped the arm voted for hundreds of records on the live clone, and each weak match collected RRF mass that a record found by meaning alone could not: a paraphrase whose target was second in the vector arm ranked it 31st overall (`docs/design.md` §16, "The vector arm"). *** ### LEXICAL\_WEIGHT ```ts const LEXICAL_WEIGHT: 1 = 1.0; ``` *** ### STOP\_WORDS ```ts const STOP_WORDS: ReadonlySet; ``` English function words dropped before indexing and before scoring a query. A short closed list: a longer one starts dropping content words (`system`, `time`) that a memory corpus searches by. *** ### VECTOR\_ARM\_LIMIT ```ts const VECTOR_ARM_LIMIT: 40 = DEFAULT_ARM_LIMIT; ``` How many records the vector arm ranks: v1's `DEFAULT_ARM_LIMIT`. Cosine gives every embedded record a score, so an uncapped arm would put a contribution on the whole corpus, most of it the noise far down a similarity list, and `arms` would name `vector` on every hit. The cap keeps the arm's vote to the records it is confident about, as v1's `LIMIT` does. *** ### VECTOR\_TEXT\_MAX\_CHARS ```ts const VECTOR_TEXT_MAX_CHARS: 1800 = 1_800; ``` The longest text one record embeds, in characters: v1's `CHUNK_MAX_CHARS` (`@memhtml/index`). Below it v1 embeds a record's whole article as chunk 0, which is the common case in a one-fact-per-file corpus, so for those records the two stores embed the same text. v2 keeps one vector per record rather than v1's chunks, so an article past the ceiling is cut, and the cut keeps the opening, where the format puts the claim. *** ### VECTOR\_WEIGHT ```ts const VECTOR_WEIGHT: 1 = 1.0; ``` The vector arm's weight: v1's (`vectorArm` in `@memhtml/index` `retrieval-sql.ts`), so a read that moves from v1 to v2 weighs a semantic match the way it did. v1 fuses four arms at 1.0 (lexical), 1.0 (vector), 0.5 (recency), and 0.4 (salience); v2 keeps lexical and vector at v1's weights and has no salience arm, and its recency is a tie-break rather than an arm. ## Functions ### advanceHead() ```ts function advanceHead( git, from, to ): Effect; ``` The version at `to`, built from `from` plus the paths the two trees differ on. Every changed path is removed from the parent's indexes and, when it still exists at `to` as a candidate, re-parsed and re-inserted. A changed path whose blob sha equals the parent record's (changed and changed back inside the range) keeps the parent's record object. Nothing else is touched, which is the structural-sharing claim the tests check by reference. #### Parameters ##### git `GitShape` ##### from [`HeadVersion`](/api/head/#headversion) ##### to `string` #### Returns `Effect`\<[`HeadVersion`](/api/head/#headversion), `InvalidMemory` \| `GitFailure`\> *** ### blobShaOf() ```ts function blobShaOf(html): string; ``` The git blob sha of `html`: sha1 over `"blob \0"` followed by the UTF-8 bytes. Computed locally rather than asked of git, because a session's overlay holds files git has not seen yet and their records need the sha a later `hash-object -w` will produce. The byte length is the UTF-8 length and not `html.length`, which is what makes multibyte content agree with git. #### Parameters ##### html `string` #### Returns `string` *** ### describeHeadGateFailure() ```ts function describeHeadGateFailure(report): string; ``` The one-line account of a failed report, in the same shape the eval gate's failure reads. #### Parameters ##### report [`HeadGateReport`](/api/head/#headgatereport) #### Returns `string` *** ### emptyVectorCache() ```ts function emptyVectorCache(space): VectorCache; ``` An empty cache in `space`, for a store that has none yet. #### Parameters ##### space [`VectorSpace`](/api/head/#vectorspace) #### Returns [`VectorCache`](/api/head/#vectorcache) *** ### fillVectors() ```ts function fillVectors(input): Effect; ``` Embed every active record of `view` whose content hash `cache` lacks, in one `embed` call (which chunks at the model's request ceiling and bounds its own concurrency), and return the cache with the new vectors added beside every vector it already held. `limit` bounds the texts one call sends, taking them in the view's record order, and `remaining` counts the ones it held back, so a caller filling in the background pays for one bounded batch at a time. Archived records are not embedded, because the arm ranks active records only. Entries for hashes the view no longer holds are kept: the cache serves every version of the store, and a record the next branch or the next rebase brings back reuses its vector rather than paying for it twice. A vector of the wrong width is refused as a defect rather than stored, because one would make the cache disagree with its own space and every cosine against it meaningless. `@memhtml/llm` already refuses such a response as a `ModelUnavailable`, so this only fires for an embedder that is not that one. #### Type Parameters ##### E `E` #### Parameters ##### input ###### cache [`VectorCache`](/api/head/#vectorcache) ###### embedder [`DocumentEmbedder`](/api/head/#documentembedder)\<`E`\> ###### limit? `number` ###### view `HeadView` #### Returns `Effect`\<[`FillResult`](/api/head/#fillresult), `E`\> *** ### gateHeads() ```ts function gateHeads( base, landed, options? ): HeadGateReport; ``` Run the gate over two versions. Pure; never throws for any pair of views. #### Parameters ##### base `HeadView` ##### landed `HeadView` ##### options? [`HeadGateOptions`](/api/head/#headgateoptions) = `{}` #### Returns [`HeadGateReport`](/api/head/#headgatereport) *** ### indexesOf() ```ts function indexesOf(view): Indexes; ``` The indexes behind any `HeadView`: reused when the view was built here, rebuilt from `records()` when a caller supplied its own (a test's `Map`-backed view). #### Parameters ##### view `HeadView` #### Returns [`Indexes`](/api/head/#indexes-1) *** ### insertRecord() ```ts function insertRecord(indexes, record): Indexes; ``` Put `record` at its path, replacing whatever was there. Archived records enter `records`, `inbound`, and `byEntity` only: the active-only indexes exist to answer "is this claim already live", and an archived file is by definition not. #### Parameters ##### indexes [`Indexes`](/api/head/#indexes-1) ##### record `MemoryRecord` #### Returns [`Indexes`](/api/head/#indexes-1) *** ### isArchivedPath() ```ts function isArchivedPath(path): boolean; ``` True when `path` sits under the archive bucket, which is what `archived` means on a record. #### Parameters ##### path `string` #### Returns `boolean` *** ### isCandidatePath() ```ts function isCandidatePath(path): boolean; ``` Whether a tree path is a memory file the head should parse. The generated `index.html` is a corpus artifact and never a memory. Anything under `.memhtml/` is state. Only `.html` blobs qualify, so `.gitkeep`, `.gitattributes`, and `sitemap.xml` are neither records nor skips: a census over the head counts eligible paths, not tree entries. #### Parameters ##### path `string` #### Returns `boolean` *** ### loadHead() ```ts function loadHead(git, sha): Effect; ``` Build the version at `sha` from a full tree read: one `ls-tree`, one `cat-file --batch`. The bulk build folds `insertRecord` over the parsed records, the same writer `advanceHead` uses, so a version built from scratch and one built by advancing agree node for node. #### Parameters ##### git `GitShape` ##### sha `string` #### Returns `Effect`\<[`HeadVersion`](/api/head/#headversion), `InvalidMemory` \| `GitFailure`\> *** ### probeQueryFor() ```ts function probeQueryFor(record): string; ``` The query a record is probed by: its title, else the first sentence of its claim, else the claim. #### Parameters ##### record `MemoryRecord` #### Returns `string` *** ### probesFor() ```ts function probesFor( base, landed, options ): readonly HeadGateProbe[]; ``` The probes: records active in both versions, ordered by a seeded digest of their path so the sample is a fixed function of `(seed, paths)` and not of insertion order, the first `count` of them. #### Parameters ##### base `HeadView` ##### landed `HeadView` ##### options ###### count `number` ###### seed `number` #### Returns readonly [`HeadGateProbe`](/api/head/#headgateprobe)[] *** ### recordFrom() ```ts function recordFrom(input): Effect; ``` Parse one file into its record. Fails with the parser's own `InvalidMemory`. #### Parameters ##### input ###### html `string` ###### path `string` #### Returns `Effect`\<`MemoryRecord`, `InvalidMemory`\> *** ### removeRecord() ```ts function removeRecord(indexes, path): Indexes; ``` Drop the record at `path` from every index. A path with no record is a no-op. #### Parameters ##### indexes [`Indexes`](/api/head/#indexes-1) ##### path `string` #### Returns [`Indexes`](/api/head/#indexes-1) *** ### scoreProbes() ```ts function scoreProbes( view, probes, k ): Scored; ``` Score every probe at one version. #### Parameters ##### view `HeadView` ##### probes readonly [`HeadGateProbe`](/api/head/#headgateprobe)[] ##### k `number` #### Returns `Scored` *** ### searchHead() ```ts function searchHead(view, input): readonly SearchHit[]; ``` #### Parameters ##### view `HeadView` ##### input [`SearchHeadInput`](/api/head/#searchheadinput) #### Returns readonly [`SearchHit`](/api/head/#searchhit)[] *** ### tokenize() ```ts function tokenize(text): readonly string[]; ``` Lowercase and split on runs of non-word characters, then drop stop words and empties. The split class is Unicode-aware (`\p{L}` and `\p{N}` plus underscore) rather than the ASCII `\W`, so `café` stays one token instead of losing its last letter. Deterministic and total. #### Parameters ##### text `string` #### Returns readonly `string`[] *** ### vectorTextOf() ```ts function vectorTextOf(record): string; ``` The text a record embeds: its article's text (`bodyText`, which opens with the claim), trimmed and cut at [VECTOR\_TEXT\_MAX\_CHARS](/api/head/#vector_text_max_chars). The title is left out on purpose. It lives in ``, outside what the content hash digests, so a vector that read it could go stale under a key that never changes: a retitled record would keep the old title's vector. BM25 still reads the title, so a title match is not lost, only not double-counted. The article is also what v1 embeds per chunk. #### Parameters ##### record `MemoryRecord` #### Returns `string` *** ### versionFromRecords() ```ts function versionFromRecords(records, sha): HeadVersion; ``` A version built from records already parsed, for a caller holding a snapshot rather than a tree. The fold is the same `insertRecord` the git load uses, so the version agrees node for node with one built from the commit, and `advanceHead` can carry it forward sharing every untouched node. A snapshot holds records only, so `skipped` is zero here: it says nothing about what the load that produced the snapshot left out, and a caller reporting the skip count of a snapshot-built version must say so. #### Parameters ##### records `Iterable`\<`MemoryRecord`\> ##### sha `string` #### Returns [`HeadVersion`](/api/head/#headversion) *** ### viewOf() ```ts function viewOf(indexes, sha): IndexedView; ``` The read surface over one `Indexes` value. #### Parameters ##### indexes [`Indexes`](/api/head/#indexes-1) ##### sha `string` \| `null` #### Returns `IndexedView` *** ### withOverlay() ```ts function withOverlay(base, ops): Effect; ``` The base with `ops` applied in order. Fails on the first op that does not parse or resolve. #### Parameters ##### base `HeadView` ##### ops readonly `OverlayOp`[] #### Returns `Effect`\<`HeadView`, `InvalidMemory`\>