Appearance
Site context & discovery
backend/src/workspace/site-context/ is the canonical read/write surface for site_pages — the workspace's page catalog. It feeds the agent and the writer; every broad read is tier-aware. Relied on by onboarding, the nightly site-refresh cron (weekly until 2026-08-18) (owned by the Pages module), the agent, and the writer's internal linking.
Two-phase discovery (page-discovery.js + store.js)
Sitemap-first, zero-AI URL discovery: robots.txt sitemap directives, sitemap-index nesting, homepage link extraction, locale-prefix detection (statistical + config-seeded), BCP-47-aware lang matching. Verified to ~28k pages in ~30s; MAX_DISCOVERY_URLS = 100k. A preview phase unblocks the onboarding wizard in seconds on huge sites before the full persist completes.
store.js persists via the clobber-safe upsert_discovered_pages RPC (pooled), cross-refs blogs (URLs already in blog_articles are dropped — exact-normalized key, see page-ownership.js for why never a locale-alias guess), guards locale mismatches, attributes pages to collections (longest-base-prefix), and runs the vanished-page lifecycle (missing_since stamping with a per-locale suspect-discovery guard + cross-host ghost stamping). The suspect guard is evidence-over-ratio since 2026-08-18: a locale whose discovery finds <50% of its tracked live rows no longer skips stamping (a genuine mass cleanup — Luniq deleted/redirected most of its 287 articles — looked exactly like a flaky sitemap and the dead rows lingered forever); it stamps a bounded batch, oldest-unseen first, capped at the realign lane's PROBE_CAP (120), and lets HTTP decide in the same run — a flaky read costs ≤120 probes and heals (200 → flags cleared), a real cleanup drains at 120 rows a night.
page-ownership.js (2026-08-16, corpus-canonical-split) is the write-time half of the ONE-OWNER-PER-URL invariant: releaseSitePageTwins(workspaceId, [{url}]) deletes an article URL's site_pages twins synchronously — called by the publish path (collections/blog.js) the moment URLs are stamped AND by the GSC cron's URL adoption (gsc/article-metrics.js, adopting a URL is claiming it — measured on Matubu: a stale publish-time URL beside the real Shopify handle), instead of waiting for the Sunday sweeps (one workspace carried 46 published-article twins). Matching is EXACT-normalized (normalizeUrlForCompare), deliberately not a locale-alias inference: every twin measured on the fleet was exact, and deleting on an inferred /x↔/en/x alias could remove a genuine page on a site whose default-locale content sits unprefixed while Orbit writes only /en/. check:data holds the invariant fleet-wide (corpus.url-owned-by-one-table, fail). GSC article discovery no longer exists as a mint path (see the gsc module page), so publishing is the only way a URL enters blog_articles.
Two-tier pages: key vs listed (selection.js)
The select_key_pages RPC scores per (workspace, locale) by page-type weight + log-scaled GSC impressions − URL depth, capped at KEY_PAGE_CAP = 20 — and, since signal plan step 4 (2026-09-01), +25 for a page the offer judge marked as selling the declared offer (null adds nothing, so the judge's absence never reshuffles a tier). No overrides since 2026-08-28: the focusPageIds pin went with the setting and the source='manual' pin with the manual-add route (zero manual rows existed on the fleet). Only KEY pages get AI spend (enrichment, embeddings, GSC inspection); "listed" rows stay tracked for free, so no page is ever lost.
The offer judge (offer-fit.js, signal plan step 4)
After each refresh (a non-fatal stage between URL realignment and the GSC sync), one batched Haiku pass judges the key + selling-shaped pages against the canonical territoryBrief — "does this page sell what the business says it sells?" — and stores the verdict on the row: sells_offer + sells_offer_reason (one line naming WHAT it sells) + sells_offer_hash + sells_offer_judged_at. Pinned by sha1/16 of the rendered brief, so an unchanged profile is never re-judged; every failure fails OPEN (unjudged stays null, a budget refusal stops the day). Deliberately a SIBLING of the agent's territory judge, not an import — the workspace ring may not reach modules/agent/, and territory.js renders while call sites judge (the same reasoning that keeps url-realign separate from collections' canonicalizer). The same stage extracts brand/product NAMES from the person's own profile text once per hash into the derived derivedBrandTerms settings key, which gsc/brand.js merges into every branded/non-branded split. Consumers: select_key_pages (+25 above), the board's lift weights (board/lift.js#offerWeight), and the card's Linked pages panel ("sells: …" through page-identity.js#resolvePaths).
Enrichment (page-enrichment.js)
One cheap Haiku pass per key page whose body CHANGED (2026-08-18: content_hash unchanged + a stored type/summary → reused, no AI call, embedding kept — what makes the nightly cadence cost cents): real title, meta, page_type, content_summary; redirect cleanup. (This is where the former page-analysis module merged.)
Refresh orchestration (site-refresh.js + runs.js)
runSiteRefresh = discovery → reconcile → selection → enrichment → URL realign → GSC sync, cross-process locked via site_context_runs (run claim / heartbeat / progress — the wizard polls this). The latest run is agent-readable too (2026-08-05): getLatestRun feeds get_workspace_profile's pipeline.siteContextRun (trigger/stage/status/counts/error/heartbeat), so a mid-rebuild, stalled, or failed catalog no longer looks like a small site.
url-realign.js (2026-08-16, corpus-canonical-split P4) is the refresh's moved/gone lane. Candidates are only what the run itself flagged — missing_since (discovery stopped seeing a tracked URL) and is_redirect (enrichment got redirected) — probed once each (cap 120/run, ≤3 same-host hops, GET with manual redirects through the shared per-host throttle throttledRawFetch (its timeoutMs option arms the abort INSIDE the throttle since 2026-08-28: a signal created before the queue wait started counting while the probe waited for its host slot or a 429 backoff and read as unconfirmed, which kept eight pruned Luniq pages alive on board cards for eleven nights) — six unthrottled workers tripped Luniq's rate limit and 29 removed articles came back unconfirmed, 2026-08-18): 200 → flags cleared; 301/308→200 → the row is realigned in place (same id, snapshots kept; url rewritten — path is a GENERATED column derived from it and is never written — collection attribution + page_type recomputed, locale follows the new URL by the catalog's own prefix map (localeForUrl: prefixed → that locale, unprefixed → the sole root locale, ambiguous → unchanged), inspection stamp cleared); 404/410 → deleted (the same confirmed-gone bar enrichment uses — listed-tier gone pages used to linger forever because only key pages got an http_status); any redirect (301/302/307/308) whose target is a LISTING — the row's collection index in any locale form, or a site/locale root (isListingTarget) → deleted too (bounced): temporary-vs-permanent only matters when the target is a content page, and a bounce to the hub is what a CMS does with a deleted post (Luniq's site 307s every removed article to /hub/articles, 2026-08-18); 302/307 to a content page, 5xx, timeouts, cross-host redirects → untouched. A move whose target is already an Orbit article's URL releases the row entirely; a target already tracked as another page deletes the probed duplicate. Enrichment no longer deletes is_redirect rows — deleting them destroyed the row's identity + snapshot history on every site redesign; it keeps setting the flag (+ redirect_target_url) as this lane's candidate signal. The articles-side twin is collections' URL canonicalizer (sitemap + bounded redirect-follow, its own nightly 04:10 cron — kept separate because a spine→module import would grow the ring baseline).
Read surface (queries.js)
Tier-aware reads for agent/writer: searchPages, retrieveRelevantCatalog, getCatalogSummary. Two additions from the 2026-08-05 tool-coverage pass:
searchPagesLoose— plain-ilike fallback over title + url when the FTS RPC comes up short: NO tier filter and NOcollection_id is nullfilter (collection-attributed pages ARE reachable), tracked AND untracked rows, each carryingis_tracked/missing_sinceso callers flag dead/untracked matches instead of hiding them. Deliberately does NOT touch the frozensearch_site_pagesRPC;search_site_pages(the agent tool) runs it as its second pass, tagging rowsmatch_type: 'substring'.getPageCorpusFacts— whole-corpus facts aboutsite_pages, the honest denominators a bounded list can't provide: total, tracked/untracked/missing counts, gsc-indexed split,by_page_type/by_locale/by_tierbreakdowns, (2026-08-23)latest_refresh— the newestsite_context_runsrow ({ trigger, status, stage, started_at, finished_at, error, counts }, viagetLatestRunso a dead 'running' row is reclaimed before it's read; null = discovery never ran): the catalog's own heartbeat, because the wizard banner and Settings could watch a failing discovery while no agent surface could — every page answer rode a possibly-stale catalog with no way to say so — and (2026-08-17)search— the site's TOTAL page search traffic fromgetPageCorpusSearchTotals(latestsite_page_snapshotsrow per page via thelatest_page_snapshotsRPC, all locales/tiers/types:pages_with_data,clicks_28d,impressions_28d,latest_snapshot_date; null on a read failure, never zero). One paginated skinny read capped at 20k rows (facts_truncatedflags the cap) + one RPC. Surfaced asget_content_inventory'spage_corpusblock, where the search totals are the denominator behind itsscope_note(a scoped page list that hides clicks says so).getPageIndexStatusCounts(2026-08-23) — the DERIVED index verdict over every tracked page, all locales and collections: thelist_site_pagesRPC'sstatusCounts+untrackedCountwithp_collection 'all'(its SQL mirrorsderiveIndexed/deriveNoindex), so the agent quotes the same indexed / not-indexed / noindex numbers the Pages filter and the Performance coverage strip show. Carried bygetPageCorpusFactsasindex_status; the rawgsc_indexed_countstays beside it, labelled raw. Null on a read failure.listKeyPages(workspaceId)/getKeyPageCitability(workspaceId)(2026-09-01, step 6) — the key pages with identity (path/url/title, for detectors minting per-page cards about them), and the citability rollup over the crawler's ownstatistics_count/quotation_count/citation_count(measured / with_statistics / with_quotations / with_citations + up to 6 thin examples) that the AI absent-prompt card attaches as facts. Null on read failure; citability also null when nothing is measured yet.listKeyPagePaths(workspaceId)(2026-09-01, signal plan step 5) — the tier='key' paths of tracked pages as a Set, null on a read failure (never an empty set: the health module's notice gate must gate nothing when the key set is unreadable). The one spine read behind "is a key page affected?".link-graph.js→getInboundLinks(workspaceId, targetPath, { limit })(2026-08-23) — who links to ONE page fromsite_page_links(the spine writes the graph instore.js#persistLinkEdges, so it owns the edge read):{ total (edges), truncated, links[{ source_page_id, path, url, title, page_type, locale }] }. Re-exported by the pages module;get_item_details' opt-inlinkssection for pages. Empty = no recorded edge, never proof of an orphan (static-HTML crawl of enriched pages only).
Support files: pool.js, embeddings.js (one of the four guarded paid clients — each batch calls ensureBudget() before the vendor, and deliberately OUTSIDE the try/catch that turns vendor failures into null vectors: a spent budget must propagate as the refusal it is, not degrade into "this page has no embedding". See Core § The spend spine).
Tables owned: site_pages, site_context_runs (cross-reads blog_articles, workspace_collections, workspaces, workspace_settings).
The internal link graph (S11, 2026-08-12)
page-enrichment.js always parsed every <a href> on every page and kept only counts — the whole graph passed through memory once per crawl and was discarded, and site_pages.inbound_links had NO writer (every consumer saw 0). Enrichment now keeps each page's internal target paths (normalized pathname, deduped, 500/page cap) and persistLinkEdges (store.js) replaces the page's edges in site_page_links (db/site-page-links-migration.sql) after the row writes, then recomputes inbound_links for the whole workspace from the graph. Best-effort: edge loss never fails a crawl. The pages module's link-equity detector is the first consumer.
Light reads for the agent (2026-08-17)
page-enrichment.js exports readPagesText(urls, { excerptChars, persistWorkspaceId }) — the same fetchOne + parseHtml the nightly enrichment uses, WITHOUT the AI classify/summarize stage — for the pages module's read_site_pages tool. articleExcerpt cuts the body text to the H1 so an excerpt judges the article, not the banner. With persistWorkspaceId, persistLightEnrichment writes the parse fields (content_text, headings, h1, dates, schema, counts, alternate_urls when hreflang was parsed) onto site_pages rows that were never enriched, stamping last_enriched_at; AI-enriched key pages are never touched, failed or redirected fetches write nothing. This is the light tier: bodies for the tail arrive on demand and stay.
The light tail pass (2026-08-17) — lightEnrichListedPages(workspaceId, { cap = LIGHT_TAIL_CAP (300), log }) runs the same fetch + parse (no AI, no vendor) over tracked non-key rows with last_enriched_at IS NULL, highest key_score first, and persists through persistLightEnrichment (URL lookups chunked at 50 — 300 long URLs in one IN() overflowed the request). Stage 2a of runSiteRefresh (after key enrichment, before URL realignment; non-fatal; counts.lightEnriched). Idempotent — the stamp makes each nightly run drain the next slice, so a 20k-page site fills over weeks. Bounded three ways because the shared fetcher serializes per host (350ms spacing + adaptive 429 backoff): the row cap, a wall-clock box (LIGHT_TAIL_TIME_MS 5 min) and an early stop after LIGHT_TAIL_MAX_429 (5) throttled answers — the result reports attempted/written/stopped (time | throttled) plus why rows were not written (failed, redirected, non_2xx, no_hreflang — a monolingual WordPress site writes bodies + dates and no alternates, which is correct). What it buys: hreflang alternates for every page, dates for staleness, a body for get_item_details. Fleet measurement 2026-08-17 (8 workspaces, Shopify/Wix/WordPress/Webflow/Drupal): Luniq 291/291 in 101s; MICE (Drupal, 10k pages) 296/300 in 249s; 2 Stream (Wix) 65/67 in 153s; Zeth 23/23 in 9s; Ledoux (WP, monolingual) 59 written, 0 hreflang; Matubu (Shopify) 300 rows → 176 × 429 in 20 min BEFORE the bounds, 110 attempted / 96 written / stopped 'throttled' in 135s after.
Locale pairs are mechanical (locale-pairs.js, 2026-08-17). Orbit articles pair by article_group_id; site pages pair by the hreflang they declare (alternate_urls). pairPagesByAlternates(pages, known) (pure, tested) treats every declared pair as an undirected edge and returns each page's connected component minus itself — symmetric (a never-fetched NL row gets its EN sibling because the EN row names it), triple-safe, normalizeUrlForCompare-keyed (www / trailing slash agnostic); a declared URL that is not a stored page keeps its URL + language with id: null. getPageLocaleVariants(workspaceId, pages) is the wrapper: one read of the workspace's alternate-bearing rows (bounded by the enriched count) + one chunked lookup for declared-but-unheld URLs. The page inventory (inventory-builders.js#buildPageInventory) attaches the result as locale_variants: [{ id, locale, url }] — absent when unpaired — so the corpus-cleanup transform groups by id ∪ locale_variants[].id instead of guessing from translated slugs (the agent's slug-token pairing on Luniq produced 23 phantom "conflicts").