Every change, in the words it was made with

Discovery probes 150 candidates a week, not 20

11:07 pm AWST · 3399bcb

The throttle was set before anyone measured what probing costs. The 2026-08-16 run measured it: 36 candidates probed in 95 seconds, about three seconds each, in a run whose mining step filed 131 new ones. Twenty a week against 131 a week is a queue that grows by design, and it had reached 98 unprobed candidates that nothing was ever going to reach.

150 keeps the step under ten minutes and drains a normal week's mining in a single pass. Seeding is untouched and stays a decision — this changes how many hosts get a verdict, not how many become sources.

Worth recording that raising this is not expected to find many dealers. Two of 59 probed candidates have ever been usable, and this morning's run was 0 of 36. The channels are all inference — links out of pages we already hold, and guessed {suburb}{marque} domains — so mining surfaces marketplaces and platform vendors rather than dealers. The queue was not the constraint; it was just the cheapest thing wrong with it.

Badge codes: the problem is real, the fix is unproven, so it does not ship

10:58 pm AWST · 0e4eec1

9,061 live listings — 5.1% — carry an MY26-shaped model year or an NX4.V4-shaped series code inside the badge field. Stripping both collapses 780 distinct badge strings across 371 cohorts, one of which carries 32 phantom trims for a single real set. Hyundai Staria sells as three different vehicles (Load van ~$54k, base ~$58k, Lounge ~$80k) and its used stock is badged Lounge, Lounge US4.V4 and Lounge US4.V4 MY27, so a Lounge subject cannot find its own kind and reads 23% low. All of that is measured and none of it is in doubt.

The fix is not shipped because the evaluation cannot demonstrate it, twice over, and both failures are the harness rather than the change:

* At the default threshold (>=120 listings, 60 models) the high-dose bucket was EMPTY. Every model small enough for badge fragmentation to hurt is excluded by the very filter that makes the fit meaningful — Staria has nowhere near 120 used listings. A clean-looking -0.15% that never exercised the motivating case, which is DATA-QUALITY 31 exactly.

* Widened to 400 models at >=40 listings it reaches that bucket: -1.38% across 592 subjects, the predicted direction. But overall lands at -0.07%, and subjects whose own badge is rewritten come out +0.14% WORSE. Against the condition restriction's -11.35% with every bucket improving, that is not a result to ship on.

The deeper limitation is that this harness models the badge *weight* (1.5 on a match, 0.8 otherwise) and not `restrictToTrim`, which is where the harm actually happens: the engine drops non-matching trims outright once eight of the subject's own exist. Fragmenting a trim into three strings defeats that restriction, and the weighting is only the consolation prize. A test that measures the consolation prize cannot price the restriction.

the small models by default, and this note so the next person starts from "model the restriction" rather than from scratch.

Operations: nothing runs on a laptop any more

10:23 pm AWST · d24e4df

The scheduling section was two architectures out of date. It described GitHub Actions deep passes at 02:07/14:07 plus a local crontab running cron-crawl.sh, and said in as many words that the Actions schedule "is the one that does not depend on any laptop being awake" — implying the other one did. Neither is true now: crawl.yml is workflow_dispatch only, and there is no Spotlot entry in any crontab or LaunchAgent on the laptop.

Everything scheduled lives in one of three places, and the doc now lists all three with their real cadences: the crawl box's four systemd timers (fast on odd hours, deep at 03:43 and 15:43, context daily, discover weekly, all Persistent=true so a missed run fires on boot), Supabase pg_cron's eight jobs, and the repository's health and CI workflows.

Recorded because someone asked the obvious question — can this run with the laptop shut — and the file that exists to answer it would have said no.

Valuation 0.12.0: a used car is not valued against new stock

10:09 pm AWST · c07c984

New-car pricing is not a price difference, it is a different transaction — recommended retail, factory incentives, dealer margin. The engine already said so, weighting a matching condition 1.4 against 0.3. That is a per-listing ratio of 4.67 and it loses to a count.

BMW X3 2026 offers 524 new listings at a $130,040 median against 21 used at $89,990. New carried 157 of the effective weight, used 29, and the answer came out $117,880 for a car its own market asks $89,990 for. Four of the ten cohorts two independent sweeps agreed were wild are a current or next model year, which is exactly where used stock is thinnest against new.

Restrict to the subject's own condition where it has at least eight comparables — the shape already used for year and trim, and it gives way last of the three, because a used car of the wrong trim or a year out is still a used car while a new one is a different market at any spec.

Held out over 72,539 used subjects across 60 models: median error -11.35%, mean -8.06%. The gradient is what makes it convincing, because it tracks the mechanism and nothing else:

new/demo share of comps subjects median error under 10% 17,349 -0.33% 10-30% 21,419 -11.84% 30-50% 24,642 -15.44% 50-70% 8,539 -17.48% over 70% 590 -21.54%

Where there is no new stock to drop it does nothing. That null result is what would have refuted this had it gone the other way, and it is the difference between this and the fingerprint dedupe, which argued just as plausibly and was worse in every bucket with a real sample.

No survivorship: it falls back rather than refusing, so all 72,539 subjects were answered by both variants — the trap recorded above the year clause, where unconditional restriction scored better partly by dropping the hard cases.

On the cohorts that motivated it: X3 2026 $117,880 -> $89,750 against an $89,990 ask, X5 2026 +26.0% -> 0.0%, Transporter 2025 +30.9% -> -6.8%. Staria 2026 barely moves, correctly — its error was an underestimate, the opposite direction, and remains open. Three sanity anchors tightened to -0.0% and none regressed.

Two caravans the exclusion list had already decided about

09:05 pm AWST · 67a7c66

The review queue should hold what a human has not ruled on. Six vehicles sat in it and two were caravan builders already listed as out of scope, reaching review only because the string missed:

A'van Owen 2012 "avan" is in the list, but makeKey strips only leading and trailing punctuation — internal characters carry meaning ("Mercedes-Benz") — so "A'van" never matched Concept Ascot 2023 "concept caravans" is in the list; this feed gives the make as bare "Concept"

Concept is matched as make+model rather than on the make alone, following the reasoning the list already records for "Golf": a make that is also an ordinary English word is exactly what a mis-parse arrives as, and excluding it outright would silently drop real cars. Verified that `Concept Coupe`, a bare `Concept`, `Golf Savannah` and `Volkswagen Golf` all stay in scope.

0173 marks the two rows already in the corpus, which resolveVehicle will not revisit because it runs only when a listing is created.

Left in the queue on purpose. Three Nissan X-Trails arrived with make "Used Vehicles" because the dealer's own URL puts that placeholder in the make slot — the extractor read the page correctly, and deciding what a placeholder make should fall back to is a change to extraction, not a list entry. It is the only placeholder make in the corpus, three listings from one dealer. And a 1973 Mercury Cougar, one car on two shopfronts: whether an American classic belongs in an Australian trade-in corpus is a scope call, and it will refuse to value for want of comparables either way.

Note that the missing odometers are zeros, not losses

08:28 pm AWST · dc1ba72

11.5% of live listings have no odometer_km and the fields blob appears to hold it anyway: 2,137 of 3,000 sampled carry km/odometer_reading/ odometer_unit. The values are 12,714 literal zeros and two real numbers. Dealers publish km: 0 on undriven stock; the parser correctly declines to read zero as a reading.

Confirmed from the other direction by condition: 29.8% of new listings lack an odometer against 3.3% of used, and used is the only population that becomes a comparable. The 11.5% headline was new cars.

No change. Recorded because the lead looks strong from key counts alone and will be picked up again — counting which keys exist is not inspecting what they hold.

Confidence is calibrated; do not fix it

07:26 pm AWST · 03f8cac

Intersecting two independent sweeps leaves 23 persistently flagged cohorts against 24 that appear in one run only and are drift. Thirteen of the 23 say high confidence at 15-30% from their own median ask, all older cars in thin cohorts — Corolla 2008 reads 29.9% low and calls it high.

Measured before touching it. Across 2,965 cohorts the label is monotonic in median gap, p90 and share-over-15%, through all four levels: high 2.0%, medium 2.7%, low 5.2%, insufficient 7.0%. On cars 2015 and older it stays monotonic and shifts up — high runs 3.8% median and 7.4% past fifteen versus 2.5% overall. So thirteen bad high-confidence cohorts is 29 of 391, which is that 7.4% tail appearing where a tail appears.

No change. The obvious next move was to make high harder to earn with age, which would have been a change to a working model justified by its own tail — the third time today an intuitive engine fix failed its measurement, after the trim-residualised year fit and the fingerprint dedupe.

Measure a comparable-dedupe change, and revert it

07:01 pm AWST · 4922795

Cross-source copies are 28.2% of live vehicles. VIN-identified ones already collapse to one comparable; 3,474 live rows are copies carrying no VIN anywhere. Folding those on make/model/year/odometer/price looked strong: the fingerprint mistakes two cars for one just 2 times in 11,526 above 20,000 km against VIN as ground truth, and one of the two is a VIN typo one character apart on an identical serial, where the fingerprint was right.

It makes valuations worse. Across 73,656 held-out subjects the median error rises 0.35%, and rises in every bucket with a real sample — +1.32% where the fold removed 5-10% of the set. 37,894 subjects degrade, 32,519 improve. The one improving bucket has 124 subjects and points the way the hypothesis predicted, which is the result not to reach for. Reverted; the engine is byte-identical to before, and the reasoning is recorded where the next person will look rather than only in this message.

The premise was wrong, not the rule. Copies do over-weight whichever dealer runs the most shopfronts — but a copy carries the same price, so it piles up at the median rather than dragging it, and removing it costs real sample. The variance added exceeds the bias removed. All that VIN-ground-truth work answered "can this be done safely?" and never asked "does it help?"

Two instruments were wrong before the right one was built, both recorded as DATA-QUALITY 35. The valuation sweep cannot see this change at all — its counters are taken upstream of the dedupe, so collapsing Grand Vitara 2017 from eight comparables to four moved none of its 3,074 numbers. And two sweeps an hour apart on a live corpus moved 1,357 estimates on their own, which read as a clean refutation (56 flagged -> 71, wild 10 -> 16) of a change that had not taken effect in a single counter.

Refresh the dedupe comment's figures; the share rose because the fix works

05:43 pm AWST · 1dc22c7

"One car, one comparable" cited 18.6% of live vehicles on two or more sources. It is now 28.2%, on 3.01 sources on average, and the HiLux example has gone from 1,283 listings over 1,065 cars to 3,254 over 1,981.

The rise is not the problem spreading. Making more VINs legible — 0154, 0164, DATA-QUALITY 29 and 34 — moves cars out of "two rows we could not tell apart" and into "one row advertised on several shopfronts", which is the state this dedupe is built to collapse. Every such move is a car that used to vote twice on a median and now votes once.

68,752 surplus rows, 36% of live listings, now collapse to one vote each. The worst single case is a 2020 Kia Seltos on thirteen Dutton domains: one VIN, one $28,990 ask, one odometer, and the same stock number in all thirteen URL slugs.

A stale figure in a comment that exists to justify a design is the kind of small dishonesty that makes the next reader distrust the rest of the file.

Integrity 17: the age of every price we serve

05:34 pm AWST · 67dbe24

A pass reports `covered` — extracted over found — and for the big yards it reads alarmingly low: tonywhitegroup 20%, pattersoncheney 15%, brisbanecars 22%. Chasing that found no parse failure. `covered` is a budget. A source with 9,130 live cars and an 1,800-page detail budget fetches a fifth of itself per pass and rotates unseen-first, so the yard is fully re-priced every five days, and crawl.ts has said so in a comment since it was written.

So the number was fine. What was missing is that nobody was watching it. Measured now, exactly rather than sampled, because listing_current already carries last_observed_at: 179,834 live priced listings, 87.4% re-confirmed inside 48 hours, 67.9% inside 24, worst case 5.9 days, nothing past ten.

None of the other twenty checks read observation age. A budget set too low, a source that quietly stops yielding, or a deep pass that stops running would stretch those numbers indefinitely while the whole suite passed and the site went on serving month-old asks as current. Stale comparables are a wrong valuation that looks exactly like a right one — there is no visible symptom, which is the argument for a check rather than an eye.

Fails at 14 days, over twice the observed worst case, so it fires on a stall and not on a slow week; warns if 48-hour coverage drops under 70%. The failure names the source holding the oldest price so it points somewhere.

Data quality log: four lessons from the engine and taxonomy work

05:24 pm AWST · 42092d2

31. The suite was blind to the subsystem it gated. A trim-residualised year-level fit was refuted by building it: zero of six target cohorts improve, four regress, Passat 2023 by -20.2%. Trim and year are collinear in exactly the cohorts that reach the level path. But the transferable part is the harness — restrictToYear fires for all twelve sanity cases and all seven anchors, so the patch left every fixture byte-identical while live on 36.6% of graded cohorts. An unmoved suite is not evidence of safety when no fixture takes the branch.

32. Two hosts, one table, an alert that cleared itself. A real failure was masked by a passing 16-check run from a box on an older commit. Runs now record whether they were scheduled and how many checks they carried.

33. Ten thousand fetches a day at sites that told us no. Marketplace index passes returned zero listings six times over, and pushed lastObs past the stall threshold while running correctly — an accurate alert with a useless cause, twice.

34. A prefix is not a truncation. Same VIN, one model a strict prefix of the other, is Pajero/Pajero Sport as often as it is a cut string. The badge is what distinguishes them; where it completes the model it is mechanically fixable, where it is a trim it needs per-car evidence.

The sanity suite grows eyes for the half it could not see

04:25 pm AWST · 38afa9a

restrictToYear fires for all twelve cases, so inside each the subject's year level and the comparable's are the same number and the year term is identically zero. The entire level / fill / yearTrust subsystem shipped untested by the suite that gates it — and that is not a theory: an adversarial review patched exactly that subsystem, left all twelve byte-identical, and was live on 36.6% of graded cohorts. "Sanity twelve unmoved" would have been reported as proof of safety. I changed this subsystem twice today under the same blindness.

Properties, not gaps, because the cohorts that reach this path are by definition thin in the subject's year, which is the same thing as having too few asks to be a yardstick. What can be asserted without one is the engine's honesty about which of three states it is in — measured, filled, or absent — and that neither of the last two is ever sold as high confidence. That is the exact defect that shipped: a 2026 Polestar 4 scoring 0.908 "high" on an estimate 29% under the only two real cars of its year, because a filled level answered levels.has(year) with yes.

The load-bearing assertion is the coverage one. If every candidate drifts into restrictToYear as the corpus grows, the section silently stops testing anything, so zero exercised cohorts is itself a failure. Today: 5 of 5, two filled, two absent, one thin-but-corrected, no case above low or medium.

Two fixes the adversarial pass endorsed, after the one it refuted

04:07 pm AWST · 19fd7b0

Both come from the same review that killed trim-residualised year levels, and both were in its "minimal fixes I would make" list.

New and demo stock no longer set the year levels. yearPriceLevels never saw `condition`, so a year whose stock happened to be new or ex-demo measured a different market from the year beside it and the whole difference was attributed to age. The 2026 Cupra Leon level came from five demo S-trims and read $35,713 against 2025's $53,560 — a -$17,847 "year step" applied to every comparable, which is what moved a $66,020 VZx to $48,161 and made it the answer. Demo excluded as well as new, which was not my first instinct and is what the evidence forced: including demos left the step exactly where it was, because a demonstrator is priced off the new car it recently was rather than the used curve it has barely joined.

This is the safe half of the fix whose other half was refuted. Trim and year are collinear, so residualising by trim regresses more cohorts than it helps. Condition is a separate dimension, and a new car is simply not evidence about a used car's age curve. The subject of a trade valuation is used; the curve should be too. Skipped when it would leave under eight points to fit, since a cohort that is genuinely all demo is better measured imperfectly than not at all.

Leon now answers $48,750 at LOW rather than $48,169 at HIGH — the estimate barely moves and the confidence tells the truth, which is the right shape for a cohort holding two physical cars.

And the GR tear is closed, in the direction the review argued for rather than the obvious one. 7 cars sat under model "GR Yaris" and 4 under "GR Corolla", both below the comparable floor and therefore unvaluable, while 32 GR-badged Yaris and 24 GR Corollas sat in the main cohorts being handled correctly. A `GR` head rule would have been the intuitive fix and is dangerous: "GR Sport" is a cosmetic pack on Landcruiser, Hilux, C-HR and RAV4, so it would mint models that do not exist, and a SQL `badge ilike 'GR%'` also catches "Grande" — 119 Kluger, 15 Prado. It is also unnecessary, because sameTrimKey already separates gr, gr gts, gr rallye and gr gt, and a GR GTS values at $57,571 against a $16k base Yaris. The trim dimension already knew; the model dimension only needed to stop disagreeing with itself. Yaris 268, Corolla 1,132, the stranded rows gone.

Sanity twelve: largest gap +6.4%, no case moved more than a rounding step.

The 102 VIN disagreements are truncations, not typos — and only 51 are fixable

03:35 pm AWST · 12c7453

I called these "probably mostly dealer typos" when I queued them. Reading them says otherwise: in every case one model is a strict prefix of the other and a VIN proves the two rows are one car. Pajero vs Pajero Sport (27), Landcruiser vs Landcruiser Prado (22), Cannon vs Cannon Alpha (22), Ora vs Ora 5, RAV vs RAV 4 Hybrid, Camry vs Camry Hybrid, Yaris vs Yaris Cross.

They split into two groups that need opposite treatment, and the difference is the whole finding.

FIXABLE — the short model is not a model at all. Toyota "RAV" carries badges that begin "4 Hybrid GX-2WD"; Toyota "RAV 4 Hybrid" carries "GX-2WD". One string, "RAV 4 Hybrid GX-2WD", cut at two different points by two parsers, and no cohort called RAV or RAV 4 Hybrid should exist beside RAV4's 1,726. Same for eleven GWM rows whose model is the make repeated with "Tank 500" in the badge. 30 split and 21 folded into one 1,766-car RAV4 cohort, hybrids keeping Hybrid in the badge where it separates cars a buyer never confuses. Idempotent on the second run.

NOT FIXABLE BY RULE — the short model is real. Landcruiser has 1,464 rows and Prado is its own model with its own market; Camry has 577 and Camry Hybrid is a different car; Pajero has 286. A prefix rule here would move 22 cars into a cohort of 1,464 and quietly wreck both. These ~90 are one car mislabelled by one source, needing per-car evidence a rule cannot supply, and they stay a warning — which is what the check that found them already called them: "taxonomy work, not a merge."

Two things the work exposed:

vin_identity_disagreements() now returns the pairs, not just the count. A check that says "102" with no route from the number to the cars cannot be acted on, and this one had sat unactionable since it was written. Its own 120s ceiling, and it still outruns that on the URL-slug half — noted rather than papered over; the merge script's own refusal log is the cheaper enumeration and is what I read.

foldableMakes was a hardcoded list of two, so a MODEL_TAIL_FOLDS entry for any other make was silently never evaluated. Toyota's did nothing at all while the sweep reported a cheerful zero. It now derives from the table, so adding a rule is enough to make it run.

The silence the stall alert could not hear

02:54 pm AWST · c733776

"crawl in flight but no observation for N minutes" only fires while a run row is open. A crawler that dies cleanly, or a box that stops, leaves no open row and raises nothing — the corpus simply stops growing while every page keeps serving numbers with confidence. The only thing that would eventually notice is the 24-hour integrity heartbeat, and a day is a long time to be certain about a stopped corpus.

So the mirror rule: nothing in flight and nothing arriving for three hours is now an alert of its own. Three because the fast pass is two-hourly and, since marketplaces came off its schedule, finishes in about thirteen minutes rather than sixty-three — an ordinary quiet stretch tops out near two hours, so three means a pass did not run or ran without writing.

Verified against the live gap that prompted it: 85 minutes between passes, correctly silent.

This is the same shape as the deep pass that never ran and the sources=0/696 that was never true — a monitor answering the question it was written to ask while the interesting failure stands just outside it.

Ten thousand fetches a day at two sites, returning nothing

02:25 pm AWST · 6c99538

market:autotrader fetched 1,500 index pages on each of six fast passes in the 24 hours to 2026-08-16 and found zero listings every time; market:carsguide did the same for 1,082. The deep pass on the same source finds 26,665. Six identical runs settle the question of whether a fast pass can read these sites, whatever the mechanism — their listings sit behind paths only the full traversal reaches.

Marketplaces are now deep-only on the schedule. A source named explicitly by id is still crawled either way, so this narrows the schedule and never the operator.

Three reasons, in the order that matters here. Ten thousand pointless requests a day at sites that already restrict automated access is not a thing to spend politeness on. It costs the fast pass most of its wall clock — autotrader alone is its last fifty minutes — so the pass runs long while producing no observations at all. And that is what tripped the 45-minute stall alert twice today: an alert correct about its symptom and useless about its cause, which is the kind that teaches a reader to stop looking.

Found by chasing that alert rather than dismissing it. The alert itself was a false positive by sixty seconds — health sampled at 06:22, the pass closed its run row at 06:23:15 — and the thing underneath it was real.

An integrity run says whether it speaks for production

01:54 pm AWST · d592896

health.ts alerted on the most recent integrity_run, which was the right question while one machine ran the suite. With a laptop and a box both filing rows it answers a different one: whichever host ran last. Today a genuine failure on 17 checks was masked four minutes later by a passing run of 16 from the box, sitting on a commit that predated the seventeenth check. The alert cleared and nothing had been fixed — the worst possible outcome for a sentinel, and I only noticed because I happened to be reading the row.

So a run now records whether the chain filed it. run-chain.sh exports SPOTLOT_CHAIN=1, check-integrity records it, and health.ts reads the latest SCHEDULED run. A developer running the suite by hand still gets their non-zero exit and their row, and can no longer silence production or alarm it.

Version skew is the other half and gets its own line rather than being longer exists, which is precisely how the mask worked. Runs record their check count, and a production run carrying fewer checks than some run has carried now says "the box is behind main".

Note for whoever reads the table next: check_count is null on every existing row and backfills itself as runs land. Deliberately not guessed — the number of checks a past run executed is not recoverable from its results column without assuming the suite never changed, which is the assumption that caused this.

The duplicate count says when it was taken

12:51 pm AWST · 8c39b15

1,006 vehicle rows retired and 1,006 listings repointed, and the check went on reporting 546 every thirty minutes — because that figure is filed by the six-hourly slow-check cron and cannot be refreshed by hand: the HTTP gateway gives up long before the fifteen-minute statement does, so the transaction rolls back on disconnect. 0169 gave the function its own ceiling and the binding constraint turned out to be the gateway, not the statement.

So the number now carries its age, and says the cron is what moves it. That is the difference between an alert meaning "go and fix this" and one meaning "this may already be fixed" — and an alert nobody can interpret is one everybody learns to close, which is the failure this session has now documented three separate times.

The merge itself verified structurally rather than on the count: across all 3,298 retired rows, zero live listings still point at one, zero retired rows hold a VIN, no dangling merge targets, no chained merges. Sanity twelve unmoved.

Worth noting where the duplicates came from: the top sources are goulburnsuzuki (247), the three Mudgee sites (134 each) and cherywindsor (48) — every one of them refit onto a better extractor this morning. The refit did not create the duplication, it made the VINs legible and so made it visible. The backlog it exposed is now drained.

Merge the cox-radius model repair

12:26 pm AWST · b210a33

A stock number and a VIN glued into the model field put 60 cars in cohorts of one, invisible to every valuation and every model page. stripStockAndVin now runs inside tidyModelString, so it applies at ingest to every platform rather than one extractor, and repairStockVinModels sweeps stored rows first in recanonicalise — before the alias, split and fold sweeps, which are all looking for a model string the reference tokens were hiding.

No re-parse needed: that session had already run its repair against production, and cox-radius now reads 0 of 1,047 live listings with a VIN in the model. What the merge adds is the ingest half, so it stays that way.

DATA-QUALITY entry numbering checked before merging, per its own warning — main ended at 29, its entry is 30, no collision this time. That check is now part of merging a parallel branch, the same way migration numbers are.

The site says when it last learned something

12:02 pm AWST · e269cda

Coverage is stated honestly on every page that carries numbers. Freshness never was, and it is the harder claim: a corpus of the right size that stopped updating on Tuesday looks identical to a live one, and a reader has no way to tell without being told. Last night made the point from the inside — a deep pass that had never run once, a fabricated sources=0/696, a stale count reported for hours, none of it visible from outside.

Deliberately not a live/dormant lamp. A crawler idle between 12-hourly deep passes is working correctly, and a red badge for the ordinary case teaches its reader to ignore the badge — the same reason health.ts stopped crying during every heavy pass. What a reader can use is when the corpus last learned anything and how often it looks.

The box writes the heartbeat because only the box knows its own schedule: the web app cannot see systemd, and a period hardcoded in the app drifts the first time a timer is retuned. run-chain.sh files one either side of a pass, twice wrapped in `|| true` — a heartbeat is a courtesy to a reader and must never change what the chain reports.

Read straight rather than off coverage_board, whose fifteen-minute refresh would render "3 minutes ago" as eighteen; a freshness claim that is itself stale is worse than none. The RPC is four rows and a max() over an indexed timestamp, with its own five-second ceiling, cached sixty seconds. The component renders nothing at all if the status cannot be read — a footer appears on every route, and half a sentence about freshness is worse than no sentence.

Seeded from crawl_run so the first render is honest rather than blank: the pass running now started 17:19 UTC and has 560 sources complete.

The model held a stock number and a VIN, and 60 cars left their cohort

11:59 am AWST · a8c0543

Cox radius detail URLs are /view/<Year>-<Make>-<Model>/<id>, and one dealer appends its own listing reference to the slug:

/view/2014-Hyundai-i30-U19524-KMHD351EMEU183127/34673102

The extractor took the year, matched the make against the canonical list, and called everything left the model. So the model was "i30 U19524 KMHD351EMEU183127" — 60 of that source's 132 live listings, and 66 vehicle rows corpus-wide.

That is worse than a wrong value. A stock number is unique per car and a VIN is unique per car in the world, so each of these was a cohort of one: the car does not appear on /model under a wrong name, it does not appear at all, and it can never be a comparable for anything including its own identical twin. Nothing looks broken from outside — a Kona is simply missing from the Kona page and no count goes red. Same shape as entry 20's badge suffix, one field over and louder, because the model is the cohort.

It had been met before and treated one car at a time. Migration 0049 carries four hand-written aliases, each naming one car's mangled model as a synonym for the real one. That is the right repair for a typo and no repair at all fresh cohort of one needing a fresh alias. Sixty had accumulated.

Stripped structurally instead — a trailing VIN, then a trailing stock number only if a VIN was found in front of it. Both tokens or neither. That the VIN is required is the whole guard, and it is measured rather than assumed: twelve stored models end in something stock-shaped with no VIN behind it, and all twelve are Avida motorhomes whose real model codes are "Bruny B7042" and "Explorer LX V5912". A rule stripping a stock-shaped tail on its own would have eaten them. No model ends in a bare VIN, and nothing in make or badge matches the pair at all.

The rule is stripStockAndVin in taxonomy.ts, beside the cars4us one it is a sibling of, so it applies to every source at ingest rather than only the one that was caught. The extractor applies the same test to the slug's tokens before they are joined, which is how the VIN survives as a value.

The VIN is kept, not discarded, and was checked before it was trusted: across 333 retained captures the slug's VIN matched the page's own vehicleIdentificationNumber 333 times and disagreed none, and the stock number matched the page's sku on all 333 likewise.

Keeping it immediately found something else. Of the 78 distinct vehicles behind those listings, 14 took their VIN and 64 were refused by vehicle_vin_key because another row already held it — all 64 held by dealer:toowoombaautomotive, a sister site on another domain and another extractor that has been reading VINs all along. That is entry 29's defect, and worth recording as an instance of it: a duplicate vehicle is not found by looking for duplicates. Nothing about either row was wrong and no string comparison would have matched them. What surfaced all 64 in one pass was an extractor learning to read an identifier it had been throwing away. Since resolved — fillVehicleBlanks now adopts into the VIN holder, all 64 pairs have collapsed, and no VIN in the corpus sits on two rows.

Re-parsing cannot repair the stored rows, by design: fillVehicleBlanks fills blanks only and excludes make, model, badge and year, because those decide which cohort a car is valued against. So the 66 rows are repaired by recanonicalise.ts, which runs first so the case, alias, split and fold sweeps all see the real model. Filtered server-side and looped until the filter comes back empty, since PostgREST caps a response at 1,000 rows and an offset walk over rows the pass is rewriting would skip some.

Re-measured against current main after a night of crawling: zero models, badges or any field carrying a VIN, and the sweep is now a no-op. npx tsc --noEmit and npm run build both clean; integrity passes 18 checks including "no VIN maps to two vehicles". The one failure, unmerged_duplicate_vehicles=546, is the documented refill from index-first listing creation and cox-radius contributes zero of it.

Fix C is refuted; the defect it was reaching for is fixed instead

11:51 am AWST · 4c844e0

The adversarial pass built a patched engine and A/B'd trim-residualised year levels over the sanity twelve, the six diagnosed cohorts and seven anchors. It improves zero of six and regresses four: Passat 2023 from 0.0% to -20.2%, CR-V 2026 from -2.4% to -15.2%. Trim and year are collinear in exactly the cohorts that reach the level path — Passat 2022 is 23 of 31 one trim, 2023 is six trims — so removing a trim offset cannot separate "2023 stock is dearer trims" from "2023 is dearer" and assigns the whole effect to trim, stripping the real year signal. The stricter reading, fitting within trim, is not computable where it is needed: zero trims have three odometer'd cars in the subject's own year across all six cohorts, so it would convert six wrong answers into six refusals. It would also key a price shift on sameTrimKey, which is demonstrably unstable across sources — the same Kamiq is "85TSI Style" at one dealer and "Style 85Tsi" at another.

The review's sharper finding is that my regression suite could not have caught any of that. restrictToYear fires for all twelve sanity cases and all seven anchors, so the year term is identically zero in every one of them; the patched engine left the suite byte-identical while being live on 36.6% of graded cohorts. I would have shipped it citing an unmoved suite as proof. That is the aggregate-hides-the-bias failure from DATA-QUALITY 26, walked back into three days later.

So, the two things the evidence actually supports:

1. The fill was laundering extrapolation into confidence. yearPriceLevels fills two years past the newest measured level, and yesterday's yearTrust asked `levels.has(year)` — which a filled level answers yes to. Three of the four worst new-generation misses kept full confidence through it: a 2026 Polestar 4 scored 0.908 "high" on an estimate 29% under the only two real 2026 cars. The function now reports which years it measured, and the discount distinguishes measured (no penalty), filled (scaled by how far past the evidence the fill reached), and absent (unchanged). Polestar 4 0.908 high -> 0.543 medium, CR-V 0.88 high -> 0.522 medium, Cupra Leon correctly untouched at 0.79 because its 2026 level was measured, RAV4 and Model 3 untouched.

2. The suite says what it cannot see, and the sweep grows the oracle that can. sanity-valuations.ts now documents its blindness to the year path rather than leaving the next reader to infer safety from silence, and sweep-valuations.ts grades a new class — confidence carried off a year level nobody measured — across the whole catalogue.

Salvage two alias guards from the parallel session before archiving it

11:32 am AWST · e6ceae4

That session measured the MODEL_ALIASES lesson harder than the N BOX note an entry in the map, and 10 of those had already come back — 128 rows — while all 9 that did reach the map have zero split rows.

Its two entries main lacked are kept: santa fe -> Santa Fe, mg s5ev -> MGS5 EV. Both cohorts read clean today (1,078 and 284 vehicles, single spelling each) and the sweep merges nothing, which is the point — these are what stop the split re-opening when a feed writes the other spelling again.

The rule the interleaved migrations earned

11:28 am AWST · 579b50e

Worktrees isolate code, not the database. One session holds the write lock on production; the others read and stage. Take migration numbers from main, not from a worktree branched yesterday. Announce corpus-changing work, because it silently invalidates another session's measurements.

Merge the duplicate-vehicle work, renumbered off a collision course

11:27 am AWST · 5b30037

The cox-radius session went well past its brief and built the thing this session had queued as the top structural item after Taj's Range Rover: a car stored as two vehicle rows, which is what let one syndicated SV count five times in a comparable set. Merging it rather than letting it sit, because it is good work and because the two branches were about to collide.

Verified before trusting, since it had already written to production — 2,292 vehicle rows merged between 01:41 and 02:45 while this session's sweeps were running. All sound: every live listing re-pointed, no VIN-holding row merged away, no dangling merge target, no chained merges. Its ingest-time path (adoptIntoVinHolder) re-points listings itself, so the caller discarding the returned id is harmless rather than the bug it looks like.

Its five migrations occupied 0154-0158, which this session had already used for the coverage board, the model splits, the Unknown-model repair and two timeouts. Renumbered to 0164-0168. The auto-merge itself was clean — the two branches touched different regions of store.ts, valuation.ts and check-integrity.ts — but that is luck, not design, and the lesson goes in the same hazard as two crawlers on one dealer.

Its new check reads a stored count filed at 02:07, mid-sweep, and so reports 357 against a live figure of effectively zero — a dry run now finds six candidates and would merge none of them (two rows that both already hold a VIN, four where the two rows disagree about the model and it correctly refuses to guess). The 06:45 cron recomputes it. Running that recount by hand first needed 0169: run_slow_integrity_checks borrowed its ceiling from the pg_cron command that calls it, so it worked on the schedule and nowhere else — a function with an invisible precondition, and the thirteenth of these tonight.

Verified on the merged tree: typecheck clean, build clean, sixteen of seventeen integrity checks pass with the seventeenth on that stale count, and the valuation sanity twelve unmoved.

Runbook: applying a migration when the Supabase MCP is not connected

10:46 am AWST · b4fea7f

The CLI authenticates from the keychain and needs no database password. `migration fetch` into a scratch workdir, drop the new file beside what it pulled, `migration up --linked`. Two differences worth knowing before relying on it: statements run outside a transaction, so `set local` silently does nothing and a failure partway leaves earlier statements applied with the version unrecorded; and the connection's timeout is about two minutes, well under the fifteen pg_cron gets.

Both were learned the hard way applying 0154 to 0158.

The duplicate-vehicle sweep runs beside discovery, where the refit is

10:45 am AWST · 59b4e54

refit-platforms.ts is what makes these visible: a source moved onto a better extractor starts reading VINs its existing listings never had, and every one that collides is a car the corpus already holds under another row. So the sweep goes directly after it, in the weekly chain rather than the crawl one.

The crawl already corrects what it can see, inline, the moment a VIN collides. This catches the rest — the VIN some platforms publish only in the detail slug, which no parsed payload carries, and whatever backlog a refit has just made legible.

spotlot-val-0.11.0: the engine stops describing work it did not do

10:43 am AWST · 2639272

Two faults found by the residual-miss diagnosis, both in the year dimension, both shipping now. The third — yearPriceLevels pooling across trims — waits on adversarial verification, since it changes how every comparable is shifted.

A. The engine claimed an adjustment it had not made. The "model year levels" note said "each comparable is also moved to <year>" whenever two levels existed and the subject had a year — including when the subject's OWN year had no level, where the shift silently no-ops and every comparable stays at its own year's price. A 2026 Polestar 2 was served at HIGH confidence with that sentence attached, valued entirely on 2023 stock, 36% under the 2026 market. An engine describing an adjustment it did not make is the same failure class as a fabricated number, and it was doing it in the workings panel that exists to make the number auditable.

The claim is now conditional on the level existing. And the dimension nobody was watching gets the discount the odometer already had: yearTrust, exp(-yearGap/2), applied when the subject's year has no measurable level. Nothing asked whether the YEAR could be corrected, and the answer is often no for exactly the cars people ask about — a year needs three odometer'd comparables, the fill reaches two years past the newest measured one, and beyond that the other three subscores stay flattered because a new-generation cohort is numerous, tightly priced and freshly listed. Polestar 2 2026 goes 0.76 "high" -> 0.17 "insufficient": the engine now declines rather than answering confidently on four-year-old stock.

B. Year and trim corrections no longer depend on the kilometre slope. The whole adjustment block was gated on `slope !== null`, so a cohort whose odometers sit too close together to fit a slope honestly discarded year levels it had already measured successfully. A 2026 Polestar 4 valued at $82,600 — exactly the 2025 median — because every comparable sat between 500 and 11,222 km, the slope refused, and the measured 2024/2025/2026 levels were computed and thrown away. Now $84,685. The three terms are independent facts about a comparable; each applies when its own inputs exist, and the per-comparable label names only the terms actually used rather than saying "odometer" because the block was once unreachable without one.

paths unchanged, RAV4 and Model 3 unchanged, C63 $96,676 high. Sanity twelve identical, band 0.0-6.3%.

The harness was grading the engine against a number it is right to disbelieve

10:35 am AWST · 5a21a6a

Both check suites counted listings where the valuation counts cars. Under syndication those are wildly different: a dealer group on six crawled sites publishes one car six times, so the "median ask" was whichever car syndicates widest, and a "12-listing cohort" can be two physical cars.

Measured on the sweep's worst outliers, by four parallel investigators over the residual misses:

Volkswagen Passat 2023 +55.4% -> -0.9% one $18,990 Traralgon car held six of eleven yardstick rows and set both the median ask AND the median odometer Skoda Kamiq 2024 -24% -> -5.1% one six-times Monte Carlo Cupra Leon 2026, Polestar 2 and 4 2026, Honda CR-V 2026 -> ungradeable: two physical cars each, defeating the eight-row floor entirely

So a large part of the "hard core" was the harness, not the engine — worth knowing before spending a week tuning a model against a broken ruler.

cohort_listings now carries vehicle_id and both suites keep one row per car, cheapest copy, matching the engine's own tiebreak (only 2.4% of multi-listed cars differ in price at all — measured earlier tonight).

The sanity twelve move by a couple of points in both directions and the band becomes 0.0-6.3%, with the header updated to say so. The yardstick got honest, not the model — the same thing the government-charges restatement did to this suite when it landed.

Real engine faults were also found and are NOT in this commit; they wait on the adversarial verification pass that ran out of credits mid-flight: year-level extrapolation into an unmeasured model year (Polestar 2 2026 claims in its own workings that comparables were "moved to 2026" when the level for 2026 does not exist and the shift silently no-ops), the year and badge level shifts being nested inside `if (slope !== null)` so a cohort with too narrow a km spread discards year corrections it successfully measured (Polestar 4 2026), and yearPriceLevels pooling across trims the same way withinYearKmSlope does — which bites harder in the levels because they shift prices directly.

One car had two vehicle rows, and the check for it could never fail

10:33 am AWST · 1333572

`check_duplicate_vins` counts vehicle rows sharing a VIN. `vehicle_vin_key` is a unique index. It has returned 0 since it was written and structurally always would, while 2,100 cars sat in the shape it cannot see: one row holding the VIN, one holding the same car with none.

`resolveVehicle` runs only when a listing is created. A listing first read from an index — which carries no VIN — gets a source-scoped fingerprint and keeps it, so the same car on a sister site whose parser could read a VIN becomes a second row. The corpus was detecting this and discarding the refuses it, and that refusal — a positive identification by the one identifier this project trusts — went to a console warning.

It cost real numbers. The sold guard shared by 0135, 0148 and 0149 is "no live listing shares this vehicle_id", so a second row hides the sister 2026-08-15 and were recorded as six sales. In comparables, BYD Sealion 6 2025 offered 35 comparables covering 30 cars, often at two different asks. Per cohort the error reaches 25%; corpus-wide it is 1.6% and invisible.

The repair is at the vehicle grain, not the source or the listing — 0082 already settled that two sites advertising one car is two real listings. `listing.vehicle_id` is repointed at the VIN holder and the emptied row is retired naming its survivor, which fixes every reader keyed on vehicle_id at once rather than asking forty SQL functions to learn a pointer. Nothing is deleted, no observation is touched, no listing changes status, and `merged_from_vehicle_id` makes it reversible.

first, both recorded in DATA-QUALITY 29 — the new check read `listing_observation.fields` directly when 74% of observations store `{}` and a hash, and a retired row still answered to its fingerprint. A third was that a periodic sweep cannot keep up at all: 289 fresh duplicates appeared in ninety minutes of ordinary crawling, so the collision is now acted on where it is found.

The health check stops failing its own physical

09:53 am AWST · 504ee75

sources=0/696 appeared in the monitor while observations flowed in the same line — an impossible fact, and a familiar one. contributing_sources timed out under concurrent crawl + sweep load and health.ts discarded the error three lines below a comment describing the previous time this exact number silently became zero. DATA-QUALITY 16, recommitted in the file that cites it.

Two layers:

- The error is read, and a count that cannot run fails the check — which under the Actions health workflow is an email, never a fabricated zero. contributing_sources joins the own-timeout club as its twelfth member.

- The nine exact counts come off the coverage board instead of the tables. The 1.3M-row observation count was blowing the ceiling at precisely the moments the database was busiest, so the health check failed loudest when everything it watches was working hardest — an alert about load wearing the costume of an alert about the corpus. A sentinel that cries during every heavy pass trains its reader to delete its emails. The board is one row under any load, 15 minutes stale at worst, and the one number that cannot tolerate staleness — the last observation's age, feeding a 45-minute stall alert — stays a direct query.

Verified under the exact load that broke it: one healthy line, 694/696.

turn_by_model joins the own-timeout club

09:47 am AWST · 1d075a8

The Most Traded panel went empty under concurrent crawl and sweep load — the RPC scans every delisted listing with a correlated not-exists and had no timeout of its own, so API renders hit the 8s ceiling and tolerant() served the blank. The pattern now has enough members that the next corpus-scanning function should be born with its timeout, not retrofitted: grep 'statement_timeout' before shipping anything that scans listing_current.

market_pulse joins the own-timeout club

09:38 am AWST · 6b98a92

Badges living as models: 4,563 vehicles folded back where they belong

09:35 am AWST · 5bd8b34

Taj retested the C63 and found the model dropdown offering both "C-class" and "C63" — and he is exactly right that it was messing up valuations: the 33 model-"C63" cars and the C-class's own C63s could never see each other as comparables, so every C63 and C43 valuation drew on half its evidence.

The third shape of model damage, inverse of the truncation split. A feed publishes the variant as the model — "C63" (34), "C200" (78), and at BMW whole variant strings: "X1 xDrive25e" (222), "X1 sDrive20i" (182), "X5 xDrive30d M Sport" (108). The X1 cohort alone was shattered across five spellings.

Foldable only where the make's own naming grammar says so, written out per make in taxonomy.ts — Audi kills any general rule on arrival, because an Audi "A4" is a model, not an A-class trim. Mercedes: an explicit class-prefix map over <letters><digits><suffix>. BMW: a base-model vocabulary whose glued tails become the badge, plus the numeric series grammar (118i, 120 Hatch, M340i all live under "1 Series"/"3 Series" with the original string as the badge head). The fold keeps every token and never stutters when the badge already carries the model.

Applied at ingest and by apply_model_folds in the recanonicalise sweep, which computes the fold per distinct model string in TypeScript and updates per string in SQL. 4,561 vehicles moved, plus 2 CLC stragglers on a second pass; a third run folds zero, which is the idempotence proof.

Measured after: X1 five spellings -> one 925-car cohort. C-class 278 -> 548 in the picker, no code-models left. C63 AMG S 2018 now answers $94,772 at HIGH confidence (the reunited cohort doubled its evidence); C43 AMG answers $54,990, the figure validated two days ago; sanity twelve in band at +4.2%; the Range Rover anchoring untouched. Catalogue board refreshed so the picker is clean now rather than at the next cycle.

Two exact-badge cars outrank nineteen wrong-trim ones

09:18 am AWST · 24730cf

A 2025 Range Rover Sport P360 Dynamic SE valued at $171,498 while its only two exact-badge comparables sat at $137,995 and $146,999 — reported by Taj within hours of the split-cohort gate shipping, and the gate never fired. Three defects composed:

1. The gate's badged path did not exist. With a badge given but below the trim floors (8 by count, 3 by divergence — this trim had 2), every piece of trim machinery was inert and the engine answered with the model year's mixture. The unbadged gate could not catch it either way: its density test looks for a hole at the mid, and this mixture is smeared — 36% of weight within 10% of the mid, D300s and D350s all through it.

The badged trigger is now disagreement, not dispersion: at least two exact-badge comparables agreeing with each other 15%+ away from the mixture's answer are the evidence, and the mixture is the noise. The estimate re-anchors on them, the range widens for the thin sample, the label caps at low, no buy price. $171,498 -> $135,775 against cars a user can see on the page.

2. The truncation repair had, correctly, made this worse. Moving the "Range" stumps into their true cohorts put five storefront copies of one $369,990 SV Edition Two into the 2025 Sport cohort — 6.2 of 24.9 effective weight, more than both exact-badge cars together, because four of the five rows carry no VIN and per-vehicle dedupe cannot merge what it cannot identify. The repair stands: the cars are real Range Rover Sports. The remaining fix is cross-source identity for unVINed syndication, which is now the top structural item, and the anchoring above neutralises its effect on badged valuations meanwhile.

3. The echo form of the truncation: badges still opening with the model's tail ("D300 Dynamic SE L461" wearing "Rover Sport " in front), whose trim keys matched nothing. apply_model_splits now strips the head where the model is already the target, driven by the same reviewed quads.

Regression battery: C63 $97,990 untouched, Patrol Warrior high-confidence untouched, Commodore unbadged still gated with spanning range, Commodore SV6 still high, Model 3 and RAV4 untouched, sanity twelve in band (largest gap +4.3%). The full-catalogue sweep restarts against this engine for the before/after.

The valuation tool learns what it was measured to not know

08:57 am AWST · 3aaa014

Four fixes from the pressure test, in user-impact order.

1. The picker stops discarding the badge. The badge select is controlled, so picking a badge and then a year whose board slice spells no such badge left the value outside the options — rendered as "any", submitted as badge=. The engine, never told the car was a C63, valued it against C200s at $33,564 with HIGHER confidence than the truthful $97,990 run. The user's chosen badge now stays in the list whatever slice is showing: it is a fact about their car, not about what one year's slice holds.

2. 726 truncated models across 16 makes are repaired, and the leak is plugged. The old jsonld extractor took one word as the model, so "Range Rover Evoque P250" was model "Range", badge "Rover Evoque P250 …" — and the "Range" cohort spanned $32,800-$369,990 across four different car lines, while 26 Teslas sat under model "Model", invisible to every Model 3 valuation. MODEL_SPLITS in taxonomy.ts names each stump and the badge head that completes it — written out per make, never derived, because Citroen's C5 proves genuinely short models exist and LDV's G10+ proves a character can be the whole difference. applied at ingest (five sources still parse via jsonld) and by apply_model_splits in the recanonicalise sweep, which moved model and badge together in one statement: a badge still reading "Rover Sport D300" would match nothing in its corrected cohort. 724+2 split, idempotent by construction, the unmatched left counted rather than guessed.

3. Eighteen spelling merges land in MODEL_ALIASES, adversarially verified. Each was upheld by a second pass told to refute it with year, fuel, badge and price evidence; the two lookalikes it protected — G10+ and RX450h+, which are different cars — stay unmerged. Most strays trace to one feed that strips hyphens; the deeper fix is the rule now written beside N BOX: a merge that lives only in a one-off migration is a merge the corpus forgets, because the recurring sweep reads MODEL_ALIASES and nothing else. Migration 0049 merged n-box once; seven fresh N-box vehicles arrived within four days.

4. spotlot-val-0.10.0: the unbadged question over a split cohort is answered as what it is. With no subject badge every piece of trim machinery was inert — the engine computed the badge divergence and never read it, so a 2017 Commodore valued as the weighted median of a bimodal mixture: a $33,330 mid at which 9.3% of comparable weight actually sits, between SV6s at $25k and SS V8s at $50k+. Now, when trim levels diverge >=1.4x AND under 20% of weight sits within 10% of the mid, the range widens to span the trim clusters, the label caps at low, the cause is named with each trim's own numbers, and the buy price is withheld — the badge decides which cluster the car is in, and a number pretending otherwise is a guess in a suit. Both conditions are load-bearing: divergence alone would have fired on five of the twelve sanity cohorts; with density it fires on zero of twelve, and on both Commodore years. Sanity suite unmoved, largest gap -4.3%.

became a 26-car cohort offering to value mystery Mazdas against each other; nulled at ingest and repaired in place), discount_by_time_on_market gets the house timeout its empty /market panel was asking for, and the day-old capture-order check gets the same — its author had just finished documenting that pattern for every other corpus-scanning function.

/value was down, over a sentence of intro copy

08:36 am AWST · 769af9e

coverage_counts computed eleven counts inline — two count(distinct url) over 200k listings, one straight count of 1.3M observations — and outgrew the API role's 8-second ceiling as the corpus grew. The await on /value sat outside tolerant(), so the valuation page answered 500 in 8.4 seconds before the form ever rendered. /market and /api had the same shape. The homepage was immune, having learned this exact lesson in its own comment; the other three pages never got it.

Found by the valuation input-path audit, which went looking for why a user's Tesla, Patrol and C63 valuations "failed at various points" — the first answer turned out to be that the front door was closed: 500 confirmed live in production during the audit, on a page whose failures were being attributed to the engine.

Three layers: - the three raw awaits are tolerant now, and the copy renders without numbers on null — a vanity count must never outrank the product; - coverage_counts stops counting and reads a one-row board refreshed by pg_cron every 15 minutes on its own 10-minute timeout, the market_pulse shape — nobody can tell a 15-minute-old observation count from a live one, and everybody can tell a page that does not load; - the board is seeded in the migration so no empty window exists.

Verified live: /value 200 in 2.1s, /market 200, /api 200, / 200.

The valuation tool gets a pressure test, and the homepage counter gets honest

08:01 am AWST · 928dd04

sweep-valuations.ts values every (make, model, year) cohort in the corpus at its own median odometer and grades each answer against the cohort's restated used asks — the question users actually put to the tool, asked 3,126 times, instead of twelve hand-picked cases. Grades: crash, refusal (no answer over 8+ used cars), wild (|gap| > 25%, the C63 class), outside the cohort's own p10..p90 band, overconfident (>15% at high/medium confidence), warn, and no_yardstick where under eight used asks exist to grade against — the pilot's flag list was mostly 2026 cohorts failing against a yardstick of one car, which is an oracle artifact, not a fault.

The sweep found its own first bug before finding any in the tool: its cohort enumeration read exactly 1,000 of 3,126 cohorts — the PostgREST response cap, silent truncation reading as completeness, the same shape this project has now hit four times. sweep_cohorts pages, and the sweep treats a page at exactly the cap as "there is more".

Fixed on the way: sanity-valuations.ts called valueCar without the internal flag, so every run of the check suite logged twelve requests as real web traffic — and market_pulse counts exactly those rows, so the "valuations" number on the live homepage was inflated by twelve per run. The 245 rows a web request with no user agent ever wrote are marked internal (every browser sends a user agent; a web request without one is a script by definition), and the suite now declares itself. The count on screen means people again.

Verified before the sweep: the three cars reported failing — Tesla Model 3, Nissan Patrol, C63 AMG S — all value cleanly today across their years, gaps 0-4.6%, so those were either pre-0.9.0 failures or input-path issues; the sweep grades the engine, and the input path is a separate question.

The sentinel moves off the box, and the ceilings get numbers

06:50 am AWST · cb13546

Audit of the VPS migration, aimed at one property: nothing vital may fail silently again. Every fault so far was invisible from inside the machine that had it — the deep pass that never ran reported success in 41ms, the pull that could not write reported a green timer, and the checks passed throughout because they live downstream of what was broken.

health.yml runs scripts/health.ts from GitHub's infrastructure every six hours. Its failure email is the alert channel — the one channel proven to reach a human, being how the truncated crawl was actually noticed. health.ts already exits non-zero on any alert, and its 24h integrity heartbeat is the wedged systemd and a skipping deep pass all surface identically as silence, then a red workflow, then an email. ~300 Actions minutes a month out of the 2,000 the retired crawl schedule freed.

The box gains a 2G swapfile (it had none, and one oversized page during a 6h pass would have been an OOM kill with no cushion), swappiness 10, persistent in fstab.

OPERATIONS gains "Where it stops scaling, measured": the deep pass runs out of TimeoutStartSec around 900-1,100 sources, the fast pass starts eating its own firings around 1,200, body storage costs ~$28/month at the one-year mark, and the observation table is a cost curve until board refreshes cross their own timeouts, which is the sign partitioning is due. Each with the sign it is approaching and the response — decisions made now, not diagnoses under pressure.

Also corrected in both docs: the 02:00-08:00 pass credited to the box was the laptop's final pass. The box was built at 12:55 UTC that day and had never completed a deep pass until tonight's, which as of this commit is mid-flight and delisting.

The deep pass had never run on the box, and said so in 41 milliseconds

06:29 am AWST · bc820aa

Two faults, both silent, found only by reading the journal after a "Crawl workflow run failed" email led there.

1. The deep pass can never start. Fast runs every two hours at :18 and takes about 63 minutes. Deep fires at :43 — inside every single fast pass. The guard against two crawlers on one host did its job and skipped, and would have skipped on every firing after that, structurally, for ever. The unit reported success in 41ms with a green timer while `delisted` sat unchanged for seven hours.

Fixed by making the deep pass wait rather than skip: a missed fast pass costs two hours, a missed deep pass costs twelve and every delisting in them. Waiting rather than re-timing, because a schedule tuned to today's fast-pass duration is a fact that expires. Bounded at two hours so a genuinely stuck crawl yields the slot instead of stacking deep passes.

2. `git pull` cannot write. ProtectSystem=strict with ReadWritePaths limited to web/logs leaves .git read-only, so every pass logged "pull failed, running 031b986" and carried on succeeding against code from before today's fixes. run-chain.sh's own header warns about exactly this — "a box pinned to whatever was cloned would break it, and the breakage would be silent" — and then the hardening pinned it anyway. node_modules for the same reason: a lockfile change triggers npm ci, which also cannot write a read-only tree. ProtectHome relaxed to read-only so npm keeps its cache instead of rebuilding it every run.

Between them: a crawler running four-day-old code and unable to retire a single car. Every check passed throughout, because every check reads the database and the database was being written to correctly — just slowly, and by the wrong pass.

Retire the Actions crawl schedule: it won the race and lost the run

06:06 am AWST · 527cb27

The workflow header said the local cron could stay because "whichever starts first takes the run, because crawl.ts refuses to start while another crawl is in flight". True when the alternative was a sleeping laptop. Against an always-on box it inverts: the race goes to whoever fires first, not whoever can finish, and this runner is capped at 15 minutes because its minutes are billed against a 2,000/month allowance.

2026-08-15, measured: the box's deep pass ran 02:00-08:00 UTC, completed 692 sources, produced 68-820 delistings an hour. At 14:07 this workflow took the next slot, hit its cap after 26 sources, and marked every traversal it had not finished incomplete — and an incomplete traversal may never infer a delisting, by design. Seven hours of zero delistings followed. Every integrity check passed throughout; the only outward sign was a "Crawl: Some jobs were not successful" email.

So it was not adding coverage on the margin. Twice a day it took the deep-pass slot away from the machine that would have done the job, and failed.

Retiring the laptop cron was necessary and not sufficient. The third scheduler nobody had thought about since it was written was still firing.

Removing the schedule would have taken RBA/CPI and FCAI with it, silently: both were gated on `github.event.schedule == '7 2,14 * * *'`, riding along with the deep pass. FuelWatch already had a timer on the box; those two had nothing, and a dataset that stops refreshing does not error — it becomes last month's number with today's date on it. spotlot-context now runs all three daily and supersedes spotlot-fuelwatch, which vps-setup.sh removes rather than leaves dormant, since both fired at 23:23 and would have fetched FuelWatch twice.

run-chain.sh pulls code but does not install systemd units, so this needs vps-setup.sh re-run on the box before the context timer exists.

reparse wrote stale prices, and invented 26 price changes doing it

03:40 am AWST · 84b5f00

A Lexus NX at Trivett went 39,990 -> 37,990 on 14 August, a real cut. On 15 August a re-parse read the 13 August capture and filed a 2,000 rise straight back up. 26 of those across the 149 sources re-parsed so far.

The cause is the order of two conditions in reparse.ts: it filtered `body_key is not null` in the query and deduplicated by URL afterwards. That reads as "the newest capture of each URL". It means "the newest capture of each URL that still has a body", and those are different rows far more often than the code's author could have known — bodies are sampled rather than kept for every fetch, and 42% of the last week's captures have no body at all. So the newest readable capture is routinely a day or two old, and re-parsing it writes that day-old price forward as the newest observation.

A sampling policy changed the meaning of a query in a tool that predated it. Nothing in reparse.ts was edited when retention became selective; the query kept returning rows and the rows kept looking like captures. Same failure as DATA-QUALITY 1 — diffing two readings that are not comparable — by a route the original fix did not cover.

It needs a dropped body AND a price that moved before the next retained capture, so 30,000 re-parsed listings produced 26: 0.2%, indistinguishable from market noise on any dashboard. Every check passed throughout. What caught it was the monitor reporting crawling=24 during the re-parse, and a crawl writing fresh prices into listings a re-parse is writing old ones into being worth thinking about before it is worth measuring.

Fixed by swapping the order: dedupe across every capture, then drop the URLs whose newest capture is unreadable. Trivett skips 8 of 1,202, 0.7%, and those get re-read by the next crawl with the current extractor.

The 26 events are deleted, matched on listing, timestamp and both price values. The stale observations stay — an observation is a true record of what a capture said, and the corpus is append-only. They leave 26 cars showing a wrong asking price until the next pass overwrites it, which is the right way

check_observation_capture_order now watches for it, as a warning rather than a failure, because a concurrent crawler can produce a benign near-tie and a check that cries wolf gets ignored.

Runbook: a source on the wrong extractor

01:17 am AWST · 56f0dab

Including the part that reads as a fault and is not: the vin fill duplicate-key warnings are two vehicle rows meeting over one physical car, which is exactly what the unique index is there to catch.

Re-read the 62,068 listings the fallback parser had already written

01:16 am AWST · a84c1af

refit-platforms.ts corrects every future pass and does nothing for the rows already written — the corpus keeps whatever the fallback managed until something re-reads the pages. Waiting for the crawl is not equivalent: the seller's listing age comes off detail pages (55% of dealer-studio detail observations carry it, against 2% of index ones), a deep pass fetches detail under a per-source cap, and it picks cars by delist-check rotation rather than by which ones are missing fields. The captures are already on disk.

Trivett, as the proof: four vehicle fields and an empty map before; after, 1,197 of 1,202 with a VIN, every one with the seller's listing age, 966 with an odometer. Nothing fetched.

Finding the sources needed one attempt and a correction. The obvious query — live listings whose `fields` is `{}` — is a jsonb comparison over 184k rows and timed out on its first and only call. The sampling trick that rescued the other corpus-wide counts is wrong here: this drives a recovery run, so a sampled answer would skip sources and the misses would be invisible, which is the exact shape of DATA-QUALITY 21. So the function is dropped rather than left to time out, and the script asks one indexed question per candidate instead — is this source's newest observation still from a jsonld parser. That is cheap, exact, and self-clearing: a source stops matching once re-parsed, so the run is safe to interrupt and safe to schedule.

Concurrency across sources, five at a time, because 62,000 separate object-storage reads is a five-to-ten hour serial run spent almost entirely waiting. Across sources only: two workers on one source would race on the same listing rows, two on different sources touch disjoint listings. Where they can still meet is the vehicle table when two yards hold one car — a race that already exists between this and the crawler, that vehicle_vin_key exists for, and that costs a VIN fill rather than an observation. Nothing here is fetched, so politeness is not the ceiling; the database is.

Written with spawn rather than spawnSync, which blocks the event loop and would have made the pool run one child at a time while claiming five.

A third of the corpus was being read by the fallback parser

12:48 am AWST · 7725aea

A source seeded without a detected platform gets `jsonld`, and that is a sensible fallback: nearly every dealer site emits a schema.org Vehicle node, so the source works immediately. It just works badly. JSON-LD carries make, model, year and price, and store.ts then prunes the blob entirely because those four are already typed columns — so a misfiled source stores four vehicle fields and an empty `fields` map, while the same page's own stock JSON sits in the retained capture holding VIN, odometer and the seller's listing age.

Across all 495 enabled jsonld sources, 440 had a better extractor already in the registry: 338 Dealer Studio, 102 i-Motor, 63,271 live listings. jsonld drops from 76,721 live listings to 13,450.

Nothing was wrong, which is why it lasted. No errors, no zero-yield sources, no stale timestamps, no failing check — they were healthy on every axis being measured. The only symptom was poverty of fields, and nothing counted fields. A fallback that fails gets fixed in a week; a fallback that quietly returns less than it could is indistinguishable from a thin source, and "that dealer doesn't publish much" is an explanation that never runs out.

More fields is not better data, so the tool is built around the thing that could actually corrupt something: identity. reparse and the crawler both key on url, so a challenger deriving a different url would create duplicates rather than enrich rows. A source moves only if the challenger returns a listing at the SAME url on every sampled page. Field count decides whether to look; identity decides whether to act. It refused none of the 440. Verifying 88 pages over 30 sources first: url identical on all 88, no nulls, one price recovered from undefined, and eight make/model disagreements, every one the new extractor being more complete (land rover range -> land rover range rover evoque). None favoured the incumbent.

Two fixes fell out of the first re-parse:

- nextjs-embedded nested its payload under fields.nextjs, one level below where every reader looks. Not a failure anywhere — the keys were stored, the queries returned nothing, and the field read as "not published". It cost all 1,965 Westside listings their days_old and price_changes_count. Flattening also lets store.ts prune properly: ~22KB an observation down to ~8KB, since the nesting had been hiding exactly the page furniture the prune list exists to refuse.

- Trivett's re-parse recovered odometer and dates for every listing and not one VIN. resolveVehicle matches by VIN first and fingerprint second, so a row created from a reading that had no VIN keeps none for ever: the next pass takes the fingerprint branch and the VIN it is holding goes nowhere. fillVehicleBlanks now fills it, in its own statement so a collision costs only the VIN and not the spec fills beside it. vehicle_vin_key is a unique index, so the database is the guard.

That last one is not a spare column: a VIN is how the corpus tells that two listings at two yards are one car, which is what stops a syndicated copy leaving one index being counted as a sale.

The sweep runs weekly beside discovery, because seeding is when the mistake is made — not in the crawl chain, where it would re-read every capture twice a day for nothing.

Photography is now refused by value rather than key name. Westside publishes primary_image_url_avif and thumbnails_avif, which the name list would have walked straight past; flattening was about to expose them.

All 15 integrity checks pass. Sanity suite unmoved, largest gap -4.9%.