# REA Top Agents Architecture And Handover

Date: `2026-05-02`

## Purpose

This is the current handover document for the private local REA top-agents system.

Start here in a new chat session. The older markdown files still matter, but they are mostly chronological build notes and forensic lessons. This file is the compact current map: what the system is, how the pieces fit, what data it owns, which guardrails must stay intact, and where to look next.

## Instructions For A New Chat

If you are starting a fresh Codex chat, give the new session this instruction:

```text
Read /Users/tr/dev/tr-wrea-local-private/docs/rea-top-agents-architecture-2026-05-02.md first. Then read the latest REA top-agents entries in /Users/tr/dev/tr-wrea-local-private/workspace-overlay/.codex/memory/current-context.md. Treat this as a local-only private WREA support system. Preserve the one-service, one-hidden-Chrome, one-queued-GraphQL-session, one-SQLite architecture unless I explicitly ask for a redesign.
```

The new session should then:

1. Check `AGENTS.md` and the workspace instructions.
2. Check `git status --short` before editing because this private repo often has unrelated local work in progress.
3. Use this handover as the current architecture map.
4. Use the older plan and investigation docs as evidence and history, not as the first source of current truth.
5. Check live `/health` and `/admin/status` with a generous timeout before declaring the service unhealthy.
6. Preserve the data-safety guardrails around canaries, zero classification, profile rotation, cooldown, and serialized browser-backed GraphQL.

## Related Docs

- [`rea-top-agents-microservice-plan-2026-04-22.md`](rea-top-agents-microservice-plan-2026-04-22.md) - original implementation plan and operating note. It explains the stable path and why the service exists.
- [`rea-top-agents-service-investigation-2026-04-22.md`](rea-top-agents-service-investigation-2026-04-22.md) - detailed investigation log. It contains the important lessons behind false-zero protection, browser profile poisoning, canaries, cooldowns, Atlas overlay import, replay, and speed validation.
- [`file-map.md`](file-map.md) - repo-level map for private helper scripts, overlays, and docs.
- [`restore-checklist.md`](restore-checklist.md) - private repo restore flow.
- [`../workspace-overlay/.codex/memory/current-context.md`](../workspace-overlay/.codex/memory/current-context.md) - short-term continuity memory. Use this for the newest operational state and recent decisions.

## Boundary

This service is local-only and private to WREA development/operator work.

It is not:

- the shared production scraper runtime
- a production WREA service
- a replacement for the shared `realestate/html/stat` production workflow
- code that belongs in `wrea`, `wrea_main`, or `scraper`

Repo and app:

- repo: `/Users/tr/dev/tr-wrea-local-private`
- app: `apps/rea-top-agents-service`
- live local URL: `http://127.0.0.1:4317`
- Tailscale/operator URL observed in use: `http://100.100.3.122:4317/admin/operator-view`

## One-Sentence Model

The system is a local REA intelligence cache: it seeds suburb identity from the production scraper database, collects REA agent/ranking/profile/listing data through one real browser-backed GraphQL session, stores the result in SQLite, and exposes local operator/admin/API views over that cache.

## Runtime Topology

```text
production scraper MySQL
  suburbs_agenthub, agents, agencies, domain_agents
        |
        v
seed / exclusions / Domain match refresh
        |
        v
local SQLite database
        ^
        |
REA GraphQL through one hidden Chrome profile
  agentSearch, agencySearch, agent(id), agentListings
        ^
        |
Node/Express service on :4317
  /admin/operator-view
  /admin/data-view
  /admin/jobs/*
  /v1/suburbs/:slug/top-agents
```

Launch and supervision:

- `scripts/install_rea_top_agents_launchd.sh` installs the launchd service for `npm start`.
- `launchd/com.tr.wrea.rea-top-agents-completion-run.plist` is a one-shot completion-run supervisor, not the main long-running service.
- The safe shape is one Node service process, one service-owned Chrome debug session, one queued GraphQL request stream, and one SQLite DB.

## Main Source Files

- `src/server.js` - Express routes, startup sequence, scheduler boot, admin job endpoints, public local API.
- `src/env.js` - local environment loading and runtime tuning from `env/.env.local`.
- `src/db.js` - SQLite schema, indexes, migrations, and connection setup.
- `src/prodScraperDb.js` - production scraper MySQL read-only bridge for suburb seed, exclusions, and Domain matches.
- `src/reaQueries.js` - REA GraphQL query documents.
- `src/browserClient.js` - hidden Chrome lifecycle, profile rotation, request queue, GraphQL fetch, cooldown/retry/recovery logic.
- `src/collector.js` - core business logic: suburb sync, canaries, zero classification, ranking writes, profile enrichment, listing ingestion, exclusions, Domain matching, status payload.
- `src/listingDates.js` - shared REA date parser for `Listed 29 Apr 2026` / `Sold 29 Apr 2026` style display text.
- `src/listingRefreshPolicy.js` - pure SOLD incremental-refresh policy helpers: cutoff calculation, order-safety checks, missing-date handling, and full-audit timing.
- `src/backfillListingDatesCli.js` - explicit one-time/maintenance CLI for normalizing existing listed/sold dates; it is not a hidden startup migration.
- `src/reportListingDuplicateCleanupCli.js` - report-only duplicate audit for listing price/date history and observation witness rows.
- `src/soldListingOrderAuditCli.js` - read-only SOLD listing order/backfill audit used to validate incremental SOLD refresh assumptions.
- `src/displayNames.js` - neutral display-name normalisation and V1 approval gate for agent/agency names.
- `src/auditDisplayNamesCli.js` - report/apply CLI for audited agent/agency display names.
- `src/scheduler.js` - serialized maintenance lanes.
- `src/operatorView.js` - HTML operator dashboard at `/admin/operator-view`.
- `src/dataBrowser.js` - read-only admin SQLite browser at `/admin/data-view`.
- `src/personClusters.js` - derived person/alias cluster read model.
- `src/docsView.js` - admin markdown preview for selected local docs.

## Data Sources

### Production Scraper MySQL

Read-only source for:

- suburb seed rows from `suburbs_agenthub`
- production blacklisted agents/agencies
- blocked email-domain agents
- Domain matching through production `domain_agents`

The service shells out to the local `mysql` client using credentials from `env/.env.local`. Do not persist secrets in tracked docs.

### Atlas CSV Overlay

Latest overlay source is selected from `exports/rea-suburbs-atlas-*.csv`.

Purpose:

- fill missing Atlas IDs
- add local overlay suburbs missing from production seed
- keep a stable local suburb universe without changing production tables

### REA Browser GraphQL

Endpoint:

- `POST /api/customer-profile-experience/graphql`

Queries used:

- `GetAgentSearch` - suburb top-agent ranking truth by Atlas ID.
- `agencySearch` - agency rankings and agency enrichment.
- `agent(id: ...)` - profile enrichment and linked salesperson IDs.
- `agentListings(ids: ..., channel: BUY|SOLD, ...)` - listing totals, listings, and samples.

Important: this behaves like browser-session GraphQL. Treat it as dependent on the service-owned real Chrome profile, not as a clean backend API.

## SQLite Data Model

Database:

- `apps/rea-top-agents-service/data/rea-top-agents.sqlite`

Core groups:

- Suburb identity:
  - `suburbs`
  - `suburb_overlay_identities`
- Ranking facts:
  - `suburb_agent_rankings`
  - `suburb_agency_rankings`
- Agent and agency identity:
  - `agents`
  - `agencies`
  - `agent_agency_history`
- Listing facts:
  - `agent_channel_stats`
  - `listings`
  - `listing_agent_links`
  - `agent_listing_memberships`
  - `listing_channel_observations`
  - `listing_price_history`
  - `listing_date_history`
- Suppression and enrichment:
  - `agent_exclusions`
  - `agency_exclusions`
  - `agent_domain_matches`
- Derived person read model:
  - `person_clusters`
  - `person_cluster_members`
  - `person_cluster_agencies`
- Operations:
  - `sync_runs`

Mental join model:

```text
suburbs
  -> suburb_agent_rankings
    -> agents
      -> agencies / agent_agency_history
      -> agent_channel_stats
      -> agent_listing_memberships
        -> listings
        -> listing_channel_observations

agents
  -> agent_exclusions
  -> agent_domain_matches
  -> person_cluster_members
    -> person_clusters
```

Important modeling notes:

- `suburb_agent_rankings` is the suburb truth from `agentSearch`.
- `agent_channel_stats` stores BUY/SOLD listing-channel freshness and totals per salesperson or linked salesperson set.
- BUY listing refresh intentionally remains full/pruning because active sale listings can disappear, be withdrawn, or move to SOLD. Do not add date-window skipping to BUY without a separate data-quality review.
- SOLD listing refresh is windowed during routine scheduler work: default `pageSize=50`, cutoff `newest known sold date - 90 days`, and monthly full SOLD audit recovery. Incremental SOLD writes update fetched rows and totals but do not prune old memberships unless all pages were fetched; forced verification, monthly audit due, unsafe sold-date ordering, or decreasing live totals fall back to full SOLD refresh.
- `agent_channel_stats` also records listing refresh telemetry: `last_full_synced_at`, `last_incremental_synced_at`, `last_refresh_mode`, `last_fetched_listing_count`, and `last_cutoff_date`.
- `listings.listing_id` is the stable REA listing identity. A BUY listing can later be the SOLD listing with the same `listing_id`; do not model this as a separate channel-specific listing identity unless live evidence proves otherwise.
- `listings.date_listed` is the normalized earliest known listed date for the listing ID, derived from REA `listing_status` text such as `Listed 29 Apr 2026`.
- `listing_channel_observations` is append-like but deduped conservatively when consecutive status, price display, listed date, and sold date are unchanged for the same listing/channel/salesperson witness.
- Linked salesperson IDs deliberately create multiple listing membership/observation witness rows for the same REA `listing_id`. That preserves alias-level listing totals and evidence. Treat these as linked-ID witness duplicates, not separate properties; use listing-ID-level price/date history or cluster/listing views when the question is property-level.
- `/admin/data-view?view=listing_property_observations` is the read-only property-level observation view. It folds linked salesperson witness rows into one listing/channel/status/price/date snapshot and exposes `witness_rows`, `salesperson_ids`, cluster count, cluster IDs, and display agent names. Use raw `listing_observations` when the alias evidence rows themselves matter.
- `listing_price_history` is change-only at listing-ID level going forward. It stores the channel and salesperson as observation context, but duplicate linked-agent witnesses should not create duplicate price-history rows when the listing price display is unchanged.
- `listing_date_history` stores distinct important listing dates at listing-ID level. Current date types are `listed_date` and `sold_date`, with `date_value` as REA display text and `date_value_iso` for sorting/filtering.
- `person_clusters` is derived from linked IDs, rankings, listing memberships, channel stats, Domain matches, and agency history. It is an admin read model and should not change public `/v1` behavior unless explicitly intended.
- Domain matching is alias-aware. `agent_domain_matches.matched_salesperson_id` distinguishes direct and inherited linked-ID matches.
- `agent_domain_matches` also stores Domain-side `domain_agent_name` and `domain_agency_name` as local evidence for display-name audits. The match identity remains exact normalized production REA agent name plus phone to Domain agent name plus phone; agency name similarity is not used to create Domain matches.
- `agents` and `agencies` have audited `display_name` fields: `display_name`, `display_name_source`, `display_name_audit_status`, `display_name_audit_rule`, `display_name_confidence`, `display_name_audit_note`, and `display_name_audited_at`.
- Public `/v1` uses `display_name` only when `display_name_audit_status = 'approved'`. Pending/review names return `null` rather than leaking a raw REA/Domain source name.
- `/v2/suburbs/:slug/agent-rankings` is the public-safe suburb ranking contract for WordPress/Laravel use. It returns only suburb identity, approved agent/agency display names, and stored `wrea_` metrics. It does not return REA IDs, source rank, Domain audit internals, listing samples, profile URLs, raw metrics, or images.
- `wrea_` public metrics are stored separately from the raw source-derived fields. Current columns include `suburb_agent_rankings.wrea_suburb_sold_count`, `suburb_agent_rankings.wrea_rank`, `agents.wrea_total_sold_count`, `agents.wrea_median_sold_price`, `agents.wrea_median_sold_days_on_market`, plus metric version/period/calculated timestamps on both tables.
- Important public-ranking nuance: the safe suburb sold count is `lead_sales + secondary_sales`. Do not use `suburb_agent_rankings.total_sales` as the suburb sold count; in live data it generally tracks the all-suburb total.
- `wrea_rank` is calculated only for public-ready rows: approved agent display name, approved agency display name, no agent/agency exclusion, and available `wrea_suburb_sold_count`. Hidden rows keep `wrea_rank = NULL`, so the public endpoint ranks visible rows starting from `1`.
- Automatic display-name approval requires Domain-supported name evidence. Agent names can be auto-approved when REA and Domain agent names agree under the current exact-name/phone Domain match evidence. Agency names can be auto-approved when Domain agency evidence agrees under loose comparison, business-suffix normalisation, connector/suffix normalisation, branch token agreement, branch-subset agreement, state-token agreement, or branch-base agreement where the REA agency name before the dash matches Domain agency evidence from matched agents. REA-only cleanup and noisy Domain mismatch rows stay in `review` and are hidden from V1 unless manually approved.
- New agent/agency inserts get a cleaned display-name candidate, but they stay in `review` when no Domain evidence is available. The explicit display-name backfill can later approve them if Domain evidence appears and agrees.
- Manual display-name decisions are made from the `Agent display audit` and `Agency display audit` reports. `Approve` writes `display_name_source = 'manual_audit'` and `display_name_audit_rule = 'manual_audit_approval'`; `Reject` writes `manual_audit_rejected` and keeps the row hidden. The optional `display_name_audit_note` records the reviewer reason/comment. The display-name backfill preserves these manual decisions. Agency audit CSV workflow is available from the agency audit/data views: export review rows with `GET /admin/data/agency-display-audit/export.csv`, fill `new_audit_status` with `approved` or `review` and `new_audit_note` in Excel, then import via `POST /admin/data/agency-display-audit/import`. CSV import uses the same manual audit fields and supports dry-run.
- Agency display names intentionally convert REA branch suffixes from dash format to bracket format. For example, `BresicWhitney - East` becomes `BresicWhitney (East)`, and `Sydney Sotheby's International Realty - Double Bay` becomes `Sydney Sotheby's International Realty (Double Bay)`. This keeps the public name neutral while preserving useful suburb/branch context.
- `/admin/data-view?view=agent_display_name_audit` and `/admin/data-view?view=agency_display_name_audit` are the first place to inspect why a V1 display name is approved or hidden for review. They include top-of-report guidance, progress summaries by status/rule, and tap/hover explanations for every audit column. Confidence is a rule score, not a statistical probability.

## Scheduler Lanes

The scheduler is enabled from `env/.env.local` and intentionally serializes browser-backed work.

Main lanes:

- exclusion refresh - import current production suppression data.
- Domain match refresh - refresh local `agent_domain_matches` from production read-only data.
- suburb refresh - refresh oldest/stale Atlas-backed suburbs.
- zero replay - recheck zero-ranking suburbs and classify confirmed true-zero rows only after canary protection.
- active-agent profiles - refresh profile and linked-ID data for ranked active agents.
- active-agent listings - refresh BUY/SOLD listing channels for active agents.
- WREA public metrics - weekly stored recalculation of deterministic `wrea_` public metrics and public-ready ranks. It is non-browser work and skips when the current weekly period is already complete.
- wide verify - random live verification of rankings and listing totals.
- profile cleanup - delete only safe old service-created auto profiles.

Admin-only maintenance lanes:

- person cluster refresh
- listing-observation dedupe

The completion supervisor in `scripts/run_rea_top_agents_completion_supervisor.mjs` was used to drive the original full baseline: suburb warming, active-agent profile enrichment, listing refresh, and final verification.

### Runtime Pause / Resume

The scheduler can now be paused without stopping the local service process. This is the preferred lightweight pause when the operator wants no background REA work and no service-owned Chrome profile, while still keeping `/admin/operator-view` available.

- `/admin/operator-view` exposes `Pause scheduler + close Chrome` and `Resume scheduler`.
- `POST /admin/scheduler/pause` clears all scheduler timers, prevents new scheduled lanes from starting, persists the paused state in `data/scheduler-runtime-state.json`, and closes the current hidden Chrome profile immediately when no scheduler job is running. If a scheduled job is already running, the close is deferred until that job finishes.
- `POST /admin/scheduler/resume` clears the persisted pause state and recreates the normal staggered timers from the existing scheduler configuration.
- `/health` reports runtime `schedulerEnabled` separately from `schedulerConfiguredEnabled`; a paused runtime can therefore show `schedulerEnabled=false` while the env-configured scheduler remains `true`.
- `/admin/status` exposes `pausedAt`, `pauseReason`, `timersActive`, `pendingBrowserClose`, and `lastBrowserClose`.
- On service restart, a persisted pause keeps the scheduler paused and attempts to close any leftover service-owned hidden Chrome process before background work can resume.
- Launchd still keeps the Node service alive so admin views and deliberate manual actions remain available. Manual admin jobs are not blocked by this pause and may reopen Chrome if deliberately run.
- The Codex heartbeat automation `rea-scheduler-health-monitor` is separate from the service scheduler. If the service is intentionally paused for a while, pause that heartbeat too so it does not report the intentional paused state as a health failure.

## Guardrails To Preserve

These are not optional implementation details. They exist because the service hit real data-safety failures during creation.

- Keep one Node service process.
- Keep one hidden/background Chrome-backed REA session.
- Keep one queued GraphQL request stream.
- Keep canaries before trusted batch writes.
- Keep positive-to-zero quarantine.
- Keep confirmed true-zero promotion behind canary-protected replay.
- Keep profile rotation/cooldown when canaries show impossible zero results or REA blocks the session.
- Keep launchd as a supervisor for the same service, not as a second worker.
- Keep old failed sync runs as history, but do not treat them alone as current health.

Do not reintroduce:

- HTML scraping fallback in this runtime
- Argonaut parsing in this runtime
- parallel browser workers
- many visible tabs or many browser sessions
- Playwright `connectOverCDP(...)` or `launchPersistentContext(...)` as a casual replacement for the current browser client
- backend-only raw GraphQL replay outside the real REA browser session
- broad deletion of browser profiles
- production writes from this private service

## Current Operator Surfaces

Useful URLs:

- `/health` - small health payload, but can be slow during synchronous SQLite-heavy maintenance.
- `/admin/status` - full JSON status payload.
- `/admin/operator` - JSON operator payload alias.
- `/admin/operator-view` - human operator dashboard.
- `/admin/listing-insights` - focused JSON payload for the operator listing panels: recent listing-level price changes, latest BUY listings, and latest SOLD listings.
- `/admin/sold-listing-order-audit` - read-only SOLD ordering/backfill audit. It reports display-order inversions, post-initial late-backdated SOLD discoveries, lookback capture buckets, and sample late rows. It does not call Chrome or production.
- `/admin/data-view` - read-only SQLite browser.
- `/v2/suburbs/:slug/agent-rankings` - public-safe suburb agent rankings with approved display names and stored `wrea_` metrics.
- `/admin/data-view?view=agents` and `/admin/data-view?view=agencies` - raw names plus audited display-name fields.
- `/admin/data-view?view=wrea_public_rankings` - internal preview of V2 public rows, raw vs `wrea_` values, public-readiness gates, and current metric period.
- `/admin/data-view?view=agent_display_name_audit` - report explaining audited agent display names, Domain evidence, rule, confidence, and V1 visibility. Includes all-column info icons, a progress summary table, manual approve/reject controls, optional audit comments, and agency Google validation links where agency context exists.
- `/admin/data-view?view=agency_display_name_audit` - report explaining audited agency display names, Domain agency evidence, rule, confidence, exclusion state, and V1 visibility. Includes all-column info icons, a progress summary table, manual approve/reject controls, optional audit comments, and a new-tab Google agency search link for public-domain validation.
- `/admin/data-view?view=agent_domain_matches` - Domain match rows plus Domain-side agent/agency names used as display-name audit evidence.
- `/admin/data-view?view=listing_property_observations` - property-level listing observation snapshots, deduped across linked salesperson witnesses.
- `/admin/docs/:docId/view` - selected local markdown preview.

Useful local checks:

```bash
curl -sS --max-time 60 http://127.0.0.1:4317/health
curl -sS --max-time 60 http://127.0.0.1:4317/admin/status
lsof -nP -iTCP:4317 -sTCP:LISTEN
tail -n 100 apps/rea-top-agents-service/logs/launchd.out.log
tail -n 100 apps/rea-top-agents-service/logs/launchd.err.log
sqlite3 apps/rea-top-agents-service/data/rea-top-agents.sqlite \
  "select id, job_type, job_key, status, started_at, completed_at from sync_runs where status='running' order by started_at desc;"
cd apps/rea-top-agents-service
npm run report:listing-duplicates
npm run report:listing-duplicates -- --listing-id 147947648
npm run report:sold-listing-order
npm run report:display-names
npm run backfill:display-names
```

Health-check nuance:

- A 5s or 10s `/health` timeout is not enough to declare the service wedged.
- During active-agent/listing work, the single Node process can spend long stretches in synchronous SQLite work.
- Retry with a longer timeout and check `sync_runs`, logs, and CPU before restarting.

## Recovery Rules

If the operator view shows suspicious zero-agent behaviour:

1. Pause manual accelerators.
2. Do not rewrite query shape first.
3. Run/inspect known-good canaries.
4. If canaries return impossible zeros, treat it as session/profile health until proven otherwise.
5. Rotate profile through the existing job path if needed.
6. Resume only after the full canary set is healthy.

If the service appears stuck:

1. Check whether one `node src/server.js` process is listening on `4317`.
2. Check `sync_runs` for a genuinely stale `running` job.
3. Check launchd logs for browser timeout/restart or SQLite migration errors.
4. Use launchd `kickstart -k` only after confirming the process is live-but-wedged, not merely busy.
5. Do not start a second manual `npm start` service on the same DB/port.

If Domain matching looks stale for a newly discovered REA agent:

1. Remember Domain matching is periodic.
2. Confirm whether the local `agents` row was created after the last Domain refresh.
3. Check linked IDs, because alias-aware Domain matching can propagate matches across person clusters.
4. Refresh Domain matches through the existing admin job if needed.

If V1 agent or agency names are missing/null:

1. Check the relevant audit report first: `/admin/data-view?view=agent_display_name_audit` or `/admin/data-view?view=agency_display_name_audit`.
2. Confirm `display_name_audit_status`; V1 intentionally hides names unless this is `approved`.
3. Refresh Domain matches if Domain-side evidence is stale, then run `npm run report:display-names`.
4. Apply the audit with `npm run backfill:display-names` only after the report counts and samples look acceptable.

## New-Session Handover Checklist

For a fresh Codex session, use this order:

1. Read this file.
2. Read the top sections and hard lessons in [`rea-top-agents-service-investigation-2026-04-22.md`](rea-top-agents-service-investigation-2026-04-22.md) if touching browser, canary, replay, zero classification, or speed.
3. Read [`rea-top-agents-microservice-plan-2026-04-22.md`](rea-top-agents-microservice-plan-2026-04-22.md) if changing operating sequence or scheduler assumptions.
4. Read the latest REA entries in [`../workspace-overlay/.codex/memory/current-context.md`](../workspace-overlay/.codex/memory/current-context.md) for current state.
5. Check live `/health` and `/admin/status` with a generous timeout.
6. Check git status before editing; this private repo often has unrelated local work in progress.
7. Keep any local-only WREA tooling/docs in this repo, not company repos.

## Current State Snapshot

As inspected on `2026-05-02`:

- Service was listening on port `4317`.
- Scheduler was enabled.
- Completion baseline state was complete from `2026-04-26T03:01:34Z`.
- Current hidden profile was healthy: `auto-profile-20260426025522915`.
- Live SQLite was about `1.1GB`.
- Approximate local data scale:
  - `17.2k` suburbs
  - `42k` agents
  - `10k` agencies
  - `599k` listings
  - `1.31M` listing observations
  - `1.31M` price-history rows
  - `133,784` listed-date history rows
  - `1,045,513` sold-date history rows
  - `40k` person clusters
- Operator listing panels are live and backed by `/admin/listing-insights`. Cold verification after the date backfill returned in about `0.24s`; full `/admin/operator-view` returned in about `0.70s`.
- Data browser includes `listing_property_observations` for property-level listing snapshots. Verification for listing `147947648` returned one row for `10 Browne Street` with `witness_rows = 3`, `salesperson_ids = 3573356,3596232,3614712`, and one cluster `3573356`. An unfiltered first page returned `603,312` property snapshots in about `10.3s`.
- Report-only duplicate audit on `2026-05-02` found old linked-ID witness duplication still present in historical tables:
  - `listing_price_history`: `296,299` duplicate listing/price groups, `712,557` excess rows under listing-level semantics.
  - `listing_date_history`: `233,709` duplicate sold-date groups, `578,915` excess rows; `listed_date` had no duplicate groups after the new backfill.
  - `listing_channel_observations`: about `182k` same-observation groups, mostly same-cluster linked aliases. These are evidence rows, not property rows.
  - No destructive cleanup has been run. The report command is `npm run report:listing-duplicates`.
- SOLD incremental refresh was added on `2026-05-05` to address weekly listing backlog without changing the one-browser architecture. Rollout audit: about `1.045M` SOLD membership/date rows, `5` display-order inversions across `5` agents, `1,195` post-initial late-backdated SOLD discoveries across `787` agents, and `1,083 / 1,195` late-backdated discoveries within the accepted `90` day lookback. Early targeted testing caught a display-order edge case for retained unfetched rows; partial SOLD refresh now pushes unfetched retained memberships after the fetched window so old evidence cannot interleave with fresh rows. Final targeted incremental test on salesperson `2683410` fetched `150` SOLD rows over `3` pages (`pageSize=50`) instead of all `4` pages, stopped at cutoff `2026-01-24`, and the audit stayed at the baseline `5` order inversions. Wide verification now persists compact listing mismatch details in `sync_runs.notes`.
- Display-name audit snapshot on `2026-05-03`: after adding conservative agency branch/state/connector/base/subset normalisation, `23,416` of `42,002` agents are approved and `4,925` of `10,273` agencies are approved. The remaining rows are `review` and return `null` in V1 unless manually approved. Agency rule counts include `2,084` branch-base agreements, `346` branch-token agreements, `102` branch-subset agreements, `21` state-token agreements, and `21` connector/suffix agreements. Branch-base approval means the REA name has a dash branch, the specific business name before the dash matches Domain agency evidence, and that Domain agency evidence came through exact agent-name/phone Domain matches. Branch-subset approval means REA lists multiple branches/suburbs and Domain evidence uses the same base name with a specific subset of those branch tokens, e.g. `Raine & Horne (Coomera - Pimpama)` versus `Raine & Horne Coomera`. The main `Agents` and `Agencies` data-browser views are sortable by column headers and now expose display-audit status filters directly; the dedicated display audit reports remain the review workflow. The admin data view menu includes `Agent display audit` and `Agency display audit`; both reports serve explanatory guidance, top summary counts/percentages, tap-friendly help on every audit column, manual approve/reject controls with optional notes, agency Google search links, and agency audit CSV export/import so operators can validate rows without reading code.
- A 10s `/health` check timed out while `active_agent_refresh` was running, but a 60s check succeeded. Treat short health timeouts as a retry signal during scheduler work.

## Useful Next Improvements

- Add a lighter-weight health endpoint or worker-state heartbeat if operator latency becomes a recurring issue during active listing refresh.
- Keep the handover file updated when the data model, scheduler lanes, or recovery rules change.
- Consider a small ER diagram export if future work on the data browser or person clusters grows more complex.
- Keep the older investigation file as evidence, not as the first thing a new session must parse.