What went wrong, and what stops it recurring

Written overnight 2026-08-10/11 while the corpus grew from ~600 to ~39,000 listings, and extended the following night at ~51,000. Every entry here is a real defect that reached the database and produced numbers that looked entirely plausible.

The reason for writing it down: nearly all of them were caught by asking is this credible? before reporting a number, not by a test. All twelve now have a structural defence (the thirteenth is recorded, not yet defended), because noticing should not be the control — and the sixth is the first that a control caught on its own, thirty minutes after it was written, with nobody looking.

If you read one entry, read the eighth. Most of the others are wrong numbers, which at least invite scrutiny. That one was an absent number with a standing explanation — "it needs more observation time" — that was plausible, comfortable and false. Nothing about a missing number asks you to check it.

Entries 9 to 12 are the same lesson arriving at a different scale: reads that returned part of the corpus and reported it as all of it. One of them broke with no code change at all — the corpus simply grew past what the query could do in the time allowed.

1Price changes diffed across two views

What happened. Six price changes appeared. All false. The crawler compares each observation with the previous one, and it was comparing a fresh reading from a listing page against a stale one from the site's search index.

Why it looked real. The prices were genuine, the direction was consistent, and the magnitudes were ordinary ($500–$2,000).

Root cause. A dealer site's search index lags its listing pages, sometimes by many hours. A Kia Carnival whose detail page recorded a price change at 02:14Z was still served at the old price by the index thirteen hours later. Both readings were honest; they were readings of different things.

Defence. Every observation records source_view (index or detail), diffs only ever compare within one view, and listing_current prefers the listing page over a newer index reading. check_cross_view_price_changes() fails CI if one ever spans the boundary.

2Delistings inferred from a stale sitemap

What happened. Thirteen cars marked sold. All false — every listing page still answered HTTP 200.

Why it looked real. They vanished from a complete sitemap traversal, which is exactly what a sold car does.

The tell. Thirteen cars at one dealer inside 3.6 hours, including new 2026 stock, most without a price ever recorded. Cars do not sell like that.

Root cause. A sitemap is a cache and lags like any other. Absence from it is not absence from the market.

Defence. A delisting now requires the listing's own page to answer 404 or

1. A candidate answering 200 stays live and is counted as

missing-from-index. check_unconfirmed_delistings() refuses any delisting without a confirming status. This is the most valuable signal in the product — a listing that disappears without a price cut is the best free proxy for sold — so it is the one most worth paying a fetch to confirm.

3Two vehicle documents on one page

What happened. A 2026 X-Trail appeared to gain $58,917 in three hours.

Root cause (partial). Dealer Studio detail pages can embed more than one vehicle document, and the extractor took whichever came first.

Defence. The document is selected by matching the vehicle id in the URL, and extraction returns null when several candidates exist and none matches. A missing observation costs one crawl cycle; a wrong one corrupts price history.

4Stock numbers reused across different cars

What happened. The same X-Trail case, whose real cause this turned out to be. Three consecutive reads of the page returned the correct $39,883, which ruled out the parser and sent the investigation to the capture log — where a Nissan Patrol and a Nissan X-Trail had been fetched seconds apart, both carrying stock number 508390 on different URLs.

Root cause. Listing matching fell back to external_id when the URL did not match, and the schema asserted (source_id, external_id) was unique — so instead of raising a conflict, it found the wrong row and wrote the Patrol's price onto the X-Trail.

Defence. The index is no longer unique, and stock-number matching requires make and model to agree. A URL is the identity of an advertisement; a stock number is a label the dealer controls.

Worth noting. The corpus showed zero reused stock numbers because the unique index made them impossible to store. The constraint was hiding the problem rather than preventing it.

5A 60% regional price gap that was fleet composition

What happened. The insight engine surfaced a striking finding: the same models asking 60% more in one state than another.

Root cause. Cohorts were keyed on make and model only, so a state stocking older, higher-kilometre examples looked like a state charging more.

Defence. Each cohort now fits its own price-by-age-and-odometer curve and compares residuals. The true spread is about 8% and not significant; a properly controlled 7% QLD/SA difference did later clear the bar, with its remaining confounders stated on screen.

6A price that was the car's own stock number

What happened. A 2025 Mazda CX-60 was recorded at $11,293,498. The health check flagged it within thirty minutes of it being written.

Root cause — not ours. The dealer's site genuinely publishes price: 11293498 in its vehicle document, alongside stocknum: "11293498". Their CMS had filled the price field with the identifier. The extractor read the correct field and the source data was wrong.

Why it is the most dangerous entry here. The bound caught this car by luck, because the stock number ran to eight digits. The identical fault on stock number 45990 produces an asking price of $45,990 — ordinary, correctly typed, impossible to distinguish from a real ask, and it would have gone silently into the comparables and moved a valuation. Every other defect in this document was detectable by looking hard at the number. This class is not.

Defence. credibleAskingPrice() in crawl/store.ts vets every price at the single point all extractors pass through, and refuses one that equals the car's own identifier whatever its magnitude — provenance, not plausibility. It compares against the identifiers the site publishes rather than the one parsed from the URL, because this car's external id was 60-11293498, which does not equal the price; URL-derived matching would have missed it. check_price_equals_identifier() asserts it in SQL, and scripts/test-price-guard.ts covers it in CI.

Refusal only governs the typed column. The raw payload still holds what the site published, so a later parser can revisit the judgement.

Cost of the rule. Zero. Across 59,837 priced observations it matched exactly one — the CX-60. A genuine Hiace asking $44,500 under stock number 544500 is untouched, which is why the comparison is equality and not a substring test.

7An out-of-scope rule that was never wired to anything

What happened. The make picker offered Yamaha, Harley-Davidson, Ducati, Jayco and Hino. Asking Spotlot to value a Yamaha would have drawn its comparables from an MT-10, a Niken and a YZF-R7.

Root cause. Three separate halves of one mechanism, none of them joined up. isNonCarMake() existed and was never called from anywhere. taxonomy_status = 'ignored' was defined, documented in migration 0016, and written only by a one-off statement covering seven rows. And no view or query ever filtered on it — so even those seven caravans stayed in the picker, the market page and the comparable pool.

Motorcycles were never classified at all, because the list only covered caravans.

Defence. Classification runs at ingest in resolveVehicle, on make and body type, and listing_current excludes ignored. One choke point rather than a rule each surface has to remember; the listing, vehicle and observation tables keep every row, so what we observed and what counts as a car stay separate questions.

Body type earns its place: BMW and Honda both sell motorcycles here, so their bikes are excluded while their cars — 465 and 546 of them — are untouched. No make-level rule can do that.

The over-reach, and how it showed. The first cut also excluded body type bus, which took out 58 vehicles: HiAces, Transits, Sprinters, a Renault Master, a VW Kombi. Australian dealers file people-mover vans under bus, and those are cars a dealer trades and values — the same commit's own comment said vans were being kept. Caught by reading the excluded list before trusting the count, which is the same discipline as every other entry here. 70 vehicles are now out of scope: 29 trucks, 24 motorcycles, 15 caravans, 2 golf carts.

8Waiting for evidence that was never going to arrive

What happened. Days to turn — the number this product exists to produce — had no value, and the stated reason was that confirmed delistings need more observation time than one night. That was wrong. More time would not have produced a single one.

The tell. 544 complete traversals had confirmed zero delistings, while every delist check logged candidates that had left the inventory index and "still answered 200". Read as "nothing has sold yet", which is not credible across 39,000 cars and half a day.

Root cause. The rule required a listing to answer 404 or 410, assuming a sold car's page disappears. Dealer Studio — 123 sources, 85% of the corpus — keeps the page and retitles it:

SOLD 2021 MG HS Essence X SAS23 in Red | Used SUV | Stock #UK15038
SOLD 2023 Toyota RAV4 GX MXAA52R in Silver | Stock #UK15164
SOLD 2022 Tesla Model Y Rear-Wheel Drive in SILVER | Stock #157636

So the rule was holding out for the weaker signal. A 404 is ambiguous — a moved page, a broken link, a migration. A title beginning SOLD is the seller stating the outcome.

Defence. Either signal confirms, and the event records which one did, so a call can be re-checked. soldMarker() matches only a title opening with SOLD or a phrase that cannot mean anything else, because a false positive here invents a sale. scripts/test-sold-marker.ts covers the real titles and the traps — "Sold Cars", "SOLD OUT", "Soldiers Point Motors", "over 3,000 cars sold", "Sold By:", and the phrase appearing only inside a <script>.

What deliberately did not change. A delisting is still never inferred. Absence from an index proves nothing on its own — defect 2 — and the evidence says keep it that way: at lismoretoyota listings leave the sitemap carrying ordinary titles, and iMotor's departed listings still say "for sale". Those stay live and unconfirmed rather than being guessed at.

Two more found in the same function. Live listings were selected unpaged, so past PostgREST's 1000-row cap a listing at a larger source was invisible to the check and could never be marked sold however long it had been gone — the third silent undercount from that cap in one day. And complete was set from !truncated alone, so an index that returned nothing counted as a full sweep, which would have made every live listing at that source a delisting candidate at once.

How it was found. scripts/probe-sold-markers.ts, kept because the answer is per-platform and the next platform will differ again: it samples listings that are live for us but gone from the source's index, and prints what their pages actually say.

9A pipeline that died silently when the corpus outgrew OFFSET

What happened. compute-insights had been failing for hours with "canceling statement due to statement timeout". The market insights on screen were simply the last set that happened to succeed — correct-looking, dated, and with nothing anywhere saying so.

The query was fine. The paging was not. Reading a large result set from PostgREST needs paging past its 1,000-row response cap, and that paging used OFFSET. OFFSET makes the database produce and discard every row before the window, so page fifty costs fifty times page one; listing_current resolves deduped fields per row, so that cost is paid again on every row thrown away. It worked at 20,000 listings and stopped at 50,000. No code changed. The corpus grew.

The defence. src/lib/paging.ts — keyset paging: "the next 1,000 rows after this id", which costs the same on page fifty as on page one because the index seeks straight to the key. All four offset-paged readers converted. Insights now reads 47,335 priced listings in 45 seconds, where it read none.

The class to watch for: a control that degrades with scale rather than failing. It gives no warning, and the day it breaks is not the day it changed.

10The rotation that kept re-fetching the same two hundred cars

What happened. Westside showed 200 extracted against nearly 2,000 cars on their website, pass after pass. The detail budget rotates never-seen listings to the front of the queue so a capped pass moves through a yard instead of re-reading the front of it. It built that order from a query of every listing at the source — unpaged, and with its error discarded.

PostgREST returned 1,000 rows and no complaint. At any yard holding more than a thousand cars, every listing past the cap was missing from the map, counted as never-seen, and sorted to the front on every pass. The same few hundred were re-fetched each time and the rest of the yard was never reached. Westside holds 1,229; Bartons 1,203; Country Cars 1,203; Brisbane Cars 1,164 — every yard big enough to need the rotation was big enough to break it.

The defence. Keyset paging again, and the error read. pageByKey throws rather than returning a partial result: a dropped error had earlier reported 287 listings at a source against a true 751, with nothing to suggest the figure was a fraction.

11An extractor that never handled the type its own header claimed

What happened. Eight sources discovered listings and extracted none — found 15, failed 15, a permanent 0% on /coverage. The JSON-LD extractor's header says it handles Vehicle/Car/Product; isVehicleNode never included Product. Several dealer platforms publish vehicle pages as a bare Product with only name, brand, image and an offer, and every one of those pages parsed to null.

The defence. Product is now accepted on evidence, never on the type alone — either a field only a vehicle has, or a brand the taxonomy recognises together with a plausible model year in the name. This matters because Product is also what an accessories shop uses for floor mats, and one site seeded the same night sells cars under a name suggesting otherwise. Tested both directions: rubber mats branded "Car Mate" and roof racks named "2022 Model" are refused; a 2025 Renault Duster and a Corolla whose marque appears only in its title are kept.

The same work caught a badge arriving as the entire listing title — "2022 Nissan X-TRAIL Ti-L e-POWER T33" — which reaches the valuation picker as an option no second listing can ever share.

12One dead URL ending a whole traversal

What happened. Both marketplaces sat at ~226 listings each. A non-200 index page broke the traversal loop outright, so everything queued behind it was abandoned: Autotrader's for-sale/nsw/page-2 returned 404 and the seven remaining state facets were never fetched. CarsGuide queues 52 facets through the same loop.

The defence. Skip the dead URL, continue the queue. The run is still recorded as partial and still refuses to infer a delisting from a partial traversal — continuing changes how much is read, never what may be concluded from it.

Adjacent, found the same way: six sources spent entire passes fetching /new-vehicles/haval-h6/ and /new-vehicles/suvs-4wds/ — model brochure and category pages — and correctly parsed them to null. The detail matcher asked for one digit after the category segment and every model name has one. It asks for four now, which a stock number or model year has and a model name does not. Measured before changing: all 957 listings held from JSON-LD sources have a four-digit run, so none were dropped.

The pattern

Every one of these produced data that was internally consistent, correctly typed, and wrong. None would have been caught by a schema, a type checker or a unit test on a fixture. What caught the first five was noticing that a number was not plausible — 13 cars sold in an afternoon, a $59k gain on a mid-size SUV, a 60% interstate gap — and then going to the source to check.

The sixth is the reason that is not good enough on its own. A stock number in a price field is invisible unless the stock number happens to be absurd, so the class cannot be found by looking at outputs, however carefully. It has to be refused at the point of entry by asking where the number came from rather than whether it looks reasonable.

So the guards that matter are the ones asserting plausibility and provenance, not shape:

cd web
npx tsx scripts/check-integrity.ts   # invariants; run before any demo
npx tsx scripts/health.ts            # one line when fine, ALERT when not

Both run after every scheduled crawl, and a failure fails the workflow. Additionally, any price movement beyond 30% or $20,000 must now be corroborated against the listing page before it is recorded at all — the X-Trail would have been caught by that alone.

13An insight that kept changing its mind

What happened. The colour-premium insight said, over one night as the corpus grew from 50,000 listings to 122,000:

The regional insight did the same and then gave up: "QLD asks 15% more than ACT", then "NSW asks 11% more than ACT", then nothing cleared the bar at all.

Why it is not caught by the existing guard. adjustForSelection multiplies the p-value by the number of candidates considered, which is the right correction for whether an extreme is real. It says nothing about which extreme. The maximum of thirteen noisy colour groups is extreme by construction, and as data arrives the argmax moves even while the p-value improves — gold to orange is not a refinement, it is a different claim.

Why it matters more than a wrong number. "Gold cars command a premium" is the kind of line a dealer repeats. If it is orange next week and teal the week after, the product was never wrong in a way anyone could catch, and was never right either. A p-value of <0.001 on the winner makes it read more trustworthy, not less.

The defence, and its limit. The fix is still a judgement about what to show, not a bug, so it has not been made. What has been built is the thing that makes the judgement possible: market_insight_history (migration 0075) records every insight that clears the bar, append-only. Until it existed the question "has this insight named the same winner twice running?" was unanswerable — compute-insights deletes and re-inserts market_insight on every run, which is correct for the live surface and destroys the claim, and market_insight_run only ever recorded how many passed, never which. The candidates are: require the same group to win across N consecutive computations before surfacing it; report the whole distribution rather than the extreme ("colour explains up to 7% of adjusted ask, widest gap orange to white"); or drop argmax insights and keep only pre-specified comparisons. The third is the most honest and the least interesting to read, which is exactly the tension.

Readings since the table was added, oldest first:

Three in a row, and the third across a corpus 8,000 listings larger and 6,400 more cars in the fit. That is more than gold ever managed — gold held for two readings and then became orange. It is not yet proof the effect is real, but it is the first time this insight's identity has been observed holding still, and it is being counted rather than remembered.

Until then, the standing caveat: an insight naming a single winner from many candidates should not be quoted as a fact about that winner. The effect size is evidence; the identity of the winner is not.

14The same car, counted twice, because the dealer was seeded twice

What happened. 132,730 live listing rows resolve to 129,190 distinct URLs. 3,540 rows — 2.67% of the corpus — are the same page stored under two source ids, and the headline count on the site includes every one of them.

Not the same car listed on two sites. The identical URL:

Every one is a single dealership seeded under more than one id, and the reasons are mundane: GWM dealers renamed from "GWM Haval", KGM is what SsangYong is called now, a group prefix appears on some entries and not others, and ftgchery is cheryferntreegully abbreviated. Discovery seeded each name it found, and nothing downstream asked whether two ids were the same yard.

Why the existing analysis missed it. docs/COVERAGE-CEILING.md measures containment — how much of one source's stock appears inside a larger one — and concluded, correctly, that duplication is a question of wasted fetches rather than of correctness, since valuations dedupe comparables by VIN. That is true for a group site restating a marque site's cars under its own URLs. It is not true of this: an identical URL under two ids inflates the count itself, and that count is published.

Why it is not just cosmetic. "132,730 listings" is the number on the home page and in every version of the README. 3,540 of them are one car wearing two hats. The rule at the top of this repository is that a thin corpus says so with the real number; a corpus that double-counts and reports the total is the same failure with a friendlier face.

Interim, applied at 07:45: migration 0078 made the published count distinct URLs, so the arithmetic stopped being wrong while the corpus still was.

Fixed that evening, in three migrations.

0082 gives the schema the word it was missing. A redundant row is neither deleted nor delisted: it takes status = 'merged' and points at the row that records the same page, and a constraint makes that pairing compulsory in both directions — a retirement names its survivor or it is not a retirement. That is the same shape as the rule that a delisting carries its evidence, and for the same reason. Deleting was never available: the observations are append-only and a better parser has to be able to re-read them. Delisting was worse — it would have manufactured 4,596 sales out of a filing error, and days-to-turn is built on exactly that signal.

A third status rather than a flag beside the other two, because every reader here names the status it wants: forty-odd SQL functions and every page ask for live or delisted, so merged drops out of all of them at once. A boolean would have needed each of them to learn about it, and the one that forgot would have gone on double-counting in silence. listing_current excludes merged rows outright, which is the choke point that matters for readers that filter nothing at all — the insight corpus pages every priced row out of that view, and a duplicate page voting twice in the price-against-age-and-kilometres fit is how entry 13's problem gets a new cause.

Which id survives was decided by evidence, not by which name reads better. The surviving row for a URL is the one whose source's base_url is the host the page is actually served from — a fact already sitting in the data, because each redundant domain redirects to the survivor's and the redundant id had been storing absolute URLs on the survivor's host all along. It cuts against taste in both directions: ftgchery beats cheryferntreegully, and mandurahgwmhaval beats mandurahhaval. For 32 URLs no id matched the host — cheryadelaide.com.au and cherynailsworth.com.au both redirect to cherymainnorth.com.au, which no source is named for — and there the earliest-observed row survives, which is arbitrary and recorded as arbitrary.

0083 ran it: 4,686 rows retired across 3,976 URLs, 24 source ids folded into 21 survivors. Before it, 156,142 live rows over 151,550 distinct URLs; immediately after, 151,604 of each. Ten live pages that only a retired id had ever seen moved to the surviving id instead of being retired — retiring a car that is still advertised is a claim, and leaving it live under a disabled source is worse, because nothing would ever look at it again. Delisted leftovers stayed put: check_delistings_from_partial_runs judges a delisting against the crawl runs of its own source, so moving a closed row to a source that never ran the traversal that closed it would break that check for nothing.

0084 is the invariant — no URL live under more than one source id — and check-integrity.ts runs it on every crawl. It will be needed: discovery will seed a yard twice again, because that is what discovery does. It now costs a failed run the next morning rather than three days of publishing a number 3% too big.

What the merge did not fix, said plainly.

• 192 vehicle rows of 94,255 now have no listing outside the merged set: the same page resolved to two vehicle records because their fingerprints differed. This merge retires listings, not vehicles, so the vehicles count still holds both.
• 42 price changes of 6,620 sit on retired rows. market_pulse and model_movers count price-change events without asking the listing's status, so those are still counted twice. It is 0.6%, and both events are real: two crawls each watched the same page change price.
• 8,701 observations belong to retired rows and stay in the total. That is honest — they are fetches that really happened — but the number counts readings, not pages, and always did.
• 90 of the retired rows were delisted, not live. Those leave listing_current, so sold counts and days-to-turn now count one exit per page instead of two. The delisting total on /coverage counts events and is unchanged: the confirmations happened, and each still carries its evidence.

It came back the same day, and the check is why we know

2026-08-13, within twelve hours of the merge. check_urls_under_two_sources() found 277 URLs live under two source ids again, and 403 rows were retired to clear it. The check earned its place faster than anything else in this file.

Where they came from. 200 rows appeared under dealer:cherymoorooka and 200 under dealer:cheryspringwood at 16:46 and 16:49 — hours after 0083 merged both into dealer:motoramachery, disabled them, and asserted neither held a live listing. Nothing has been written to either since, so it was one run rather than a leak: all three ids point at motoramachery's own sitemap, so whatever walked it created the same page three times.

Why three defences all missed it. crawl.ts only walks sources where enabled is true. 0082's constraint refuses to enable a source that still names its survivor. Both are about reaching a source through the scheduled path — and neither stands between a source id and observeListing, which is the only function that creates a listing row. Anything holding an id could write to it: a targeted crawl, a reparse, a backfill.

Defence. observeListing now refuses to create a listing whose source carries merged_into_source_id, naming the survivor in the error. Creation only: re-observing an existing merged row already follows its pointer to the survivor, which is the right reading of the same page.

The distinction worth keeping: 0084's check is a monitor and this is the defence. A check that reports the same breach every morning is not a control — it is a subscription to the problem. This is the third time in this file that a rule existed, was correct, and was never in the path that mattered (defect 7, the stale-board alert in health.ts, and now this).

Still open, deliberately. The guard lives in application code, so a direct insert into listing bypasses it. The structural version is a trigger, and it is not written yet: three sessions were mid-flight against this database at the time, and adding a migration would have collided with a numbering reconciliation already in progress. It is the right next step once that settles.

15Two days of daily rollup are missing, and cannot be recovered

What happened. daily_model_rollup and daily_market_rollup hold one row per model per day, written in-database by pg_cron at 15:55 UTC. The job failed on 2026-08-11 and 2026-08-12 — both times canceling statement due to statement timeout, both stopping at exactly 00:02:00. 2026-08-12 has no rollup at all and never will.

write_daily_rollup joins listing_event to listing_current and scans it again for the day's delistings. At 137,000 live listings that work outgrew the cron role's 2-minute default. Measured after the fix: 188 seconds.

Why nothing noticed for two days. A cron job that fails silently is indistinguishable from one with nothing to do. What caught it was migration 0077's stale-board check, which alerts when a board has not been refreshed inside its own limit — the tables were 36.3 hours old. Without that check this would have been found whenever somebody next looked at a four-week range label and wondered why it stopped moving.

The fix, and the one it should have been part of. Migration 0073 gave the depth-boards job a 15-minute ceiling for exactly this reason, and left daily-rollup on the default. 0079 does the same for it. Two jobs, one shape of problem, fixed six days apart because the first fix was scoped to the job that was failing rather than to the class.

Why 2026-08-12 is not being backfilled. write_daily_rollup takes a date, which makes backfilling look easy and safe. It is not. Only cuts, rises and confirmed_exits are filtered by that date; live, vehicles, the used/new/demo counts and every ask-price quantile read current state. Running it for 2026-08-12 today would write today's market under yesterday's date and label it history. The row would look perfect and be a lie, which is worse than the gap. Consumers of this table should expect 2026-08-12 to be absent rather than wrong.

16The same bug four times in one day: `const { data } = await`

Not a defect in the data. A defect in how this codebase asks for it, which produced four wrong answers on 2026-08-13 alone:

Three share one line of code:

const { data } = await client.from(…)…;   // error is discarded

supabase-js returns { data, error } and resolves either way. Destructure only data and a failed query is indistinguishable from an empty result — and because empty results are normal, every caller already has a sensible-looking branch for it. The failure does not look like a failure. It looks like a fact.

Why it keeps happening here specifically. Both halves of this codebase are built on "absence is meaningful": a source with no sitemap, a listing with no previous observation, a vehicle with no VIN. That is the right model, and it makes the empty-result branch the well-trodden one. A dropped error walks straight down it.

What is already defended. The ingest path's identity lookups (store.ts) drop errors too, and the damage there is bounded by unique indexes on vehicle.vin and vehicle.fingerprint: a swallowed error leads to an insert that violates a constraint and throws, rather than a duplicate vehicle. That is luck rather than design, but it is real, and it is why the crawler was not rewritten in a hurry on a day when three other things were.

The rule, stated once. Read the error whenever "nothing" and "could not ask" would lead to different behaviour. Where they lead to the same behaviour, say so in a comment, so the next reader knows it was decided rather than skipped. pageByKey() in src/lib/paging.ts exists because of exactly this class and has the discipline built in — it never swallows.

Not yet swept. 52 call sites destructure only data. Fixing them all is a day's careful work across the crawler's core, and doing it in a hurry to close an entry would be the wrong trade on a live corpus. The two that were producing wrong answers on the day were fixed; the rest are listed by grep -rn "const { data" src scripts | grep -v error.

Two more, that evening, and the second one is the worst yet. Found while checking whether the duplicate-source merge had broken anything — which it had not:

health.ts counted with const { count: n } = await q and returned n ?? 0, so a cancelled count became a confident zero in a one-line report designed to be skimmed. It read as the entire corpus having been delisted overnight; the next two runs printed the real number, which is how a transient failure looks when nobody can tell it from data.

The second is the one that matters. The integrity suite was reporting a pass for a check that could not run. check_price_equals_identifier() reads fields for every priced observation — 3.7 million buffer hits, 78 seconds — and had crossed the API role's 8-second ceiling as the corpus grew. The RPC returned an error, the caller destructured only data, null counted as zero, and the line printed green. This file's own rule, written the same day about four other cases, is that a check that cannot run has to say so.

Defence. Every check in check-integrity.ts and every count in health.ts now reads its error and throws, so a check that cannot run fails the workflow instead of decorating it. Two helpers — rpcCount and rowCount — make that the only way to write one. 0085 gave the two slowest functions their own 60-second ceilings, which is measurably real (the cancellation moved from 8,171ms to 60,223ms) and was still not enough for a 78-second query, so 0086 moved that invariant off the request path: computed 6-hourly by pg_cron under the 15-minute ceiling, stored with its computed_at, and the suite fails on a stale row as well as on a non-zero count. First honest run: 0 across 485,798 observations.

The remaining 53 call sites are still unswept. The two files that gate the crawl are not among them any more.

The pattern behind 9 to 12

The first eight were wrong numbers. These four are absences that looked like completeness — a query that returned less than it should and said nothing about it. Three were the same root cause in different clothes: a result set truncated by a cap or a timeout, with the error dropped or never checked.

A wrong number invites scrutiny. A number computed from 2% of the corpus looks exactly like a number computed from all of it, and a source stuck at 200 looks like a small dealer. So the rule that came out of this night:

Never let a read fail quietly. Read every error. Assume every result set is capped until it is paged. Prefer paging that does not get slower as the corpus grows, because the alternative fails on a schedule nobody set.

17A guard that measured its own assumption, and then the assumption moved

The 00:36 monitor alerted: 25 delisting(s) from an incomplete crawl. That is the one alert this project treats as never-ignorable, because the invariant behind it — a delisting may only be inferred from a fully-traversed index — is what stops a site outage becoming a hundred fabricated sales.

The delistings were real. Every candidate had been confirmed by fetching its own page, and those pages answer 200 with the dealer's own title reading SOLD 2026 Nissan Patrol Ti Y62 in White | … | Country Cars. Nothing was fabricated and nothing needed reverting.

The run row was wrong, and the cause was a change made four hours earlier in the same session. closeAbandonedRuns() reaps open runs older than two hours, justified by a comment that measured exactly the thing it guarded:

no single source takes anywhere near this: the largest budget is ~250 detail fetches at ~9s, about 40 minutes

Fitting detail budgets to real stock raised brisbanecars to 8,150 pages and countrycars to 6,200 — three to four and a half hours of legitimate fetching. The reaper then closed both runs mid-traversal, stamping them finished, incomplete, zero pages, while they were working normally. Both later finished correctly, which is why the alert cleared on its own and why the corpus was never affected.

The false alarm was the visible half. The same constant gates anotherCrawlIsRunning(). A run dismissed as stale stops being a claim on its source, so a second pass could have started against hosts the first was still fetching — doubling the request rate on every shared host, which is the one thing SPEC.md does not permit. The cron wrapper's pgrep would have caught it on this machine; the database claim exists for the case where it would not.

What was actually wrong

Not the number, the signal. Age cannot distinguish a crawl that died from a crawl that is slow, and "slow" had just been redefined by a config change in another file. A crawler writes observations continuously, so its liveness is observable rather than inferrable: a run is abandoned only if it is old and nothing has been written for twenty minutes.

Checked corpus-wide rather than per source, because per source it would still reap a run whose pages happen to be 404ing for a stretch — that source records nothing while its process is perfectly alive. Any recent observation proves a crawler exists; if one exists, its rows are not abandoned. An unreadable check returns "not quiet", because unknown must never license reaping.

A first attempt derived the window from the largest configured budget. It worked and was the wrong trade: at 8,150 pages it computes to nineteen hours, so a crawl that genuinely died would block every scheduled pass for most of a day — the failure the window exists to prevent, relocated rather than removed.

The pattern

Entries 9–12 were about reads that failed quietly. This one is different: a guard that was correct, well-commented, and documented its own justification in terms of a value living somewhere else. The comment was not wrong when it was written; it was falsified by a change in another file that had no reason to know the comment existed.

So: when a guard's justification quotes a number it does not own, it will eventually be guarding something else. Prefer a signal the guard can observe directly — here, "is anything being written" — over one it has to assume. And when a change invalidates a premise elsewhere, the alert it produces may be the only notice you get; dealer:peterwarren was already open past the window and would have been the third source reaped mid-work.

18Two crawlers on one host, and the host said 503

The concurrency guard described in entry 17 as a hypothetical had already failed, and the evidence was sitting in the same pass.

market:autotrader has two overlapping run rows on 2026-08-13: one opened at 12:40 UTC and still running, another opened at 21:10 UTC. Nine hours apart, same source, same host. Late in that overlap the crawl log records

~ www.autotrader.com.au: HTTP 503 — slowing to 2.0s between requests

which is the site declining to serve at the rate it was being asked to. Per-host politeness is enforced within a process, so two crawlers against one host is two request streams and the pacing each of them thinks it is keeping is only half the truth. SPEC.md does not permit that.

It was exactly one source, not dozens — a second full pass would have overlapped everything, so this was a targeted run against autotrader, started while the first was still fetching it.

Why the guard let it through

anotherCrawlIsRunning() looked for open run rows newer than STALE_RUN_HOURS, which was two. The 12:40 run was already eight hours old, so it fell outside the window and stopped counting as a claim on the source. The guard was not bypassed; it was asked the wrong question — "has a crawl started recently" rather than "is a crawl still working".

The window was two hours because of a comment measuring the budgets it guarded: "no single source takes anywhere near this: the largest budget is ~250 detail fetches at ~9s, about 40 minutes." Budgets are now up to 8,000 pages, and autotrader answers in about four seconds a page.

Fixed

An open run row is a claim regardless of age (72h lookback), and rows are reaped on liveness rather than age — old and nothing written anywhere for twenty minutes, with a twelve-hour hard ceiling so a dead pass cannot deadlock a chained one. A long-running source therefore keeps its claim for as long as it is actually working, which is the whole point.

Also bounded the bite: autotrader's budget went 8,000 → 3,000, so one slow marketplace stops gating a pass over 690 dealer sources. See OPERATIONS, "One source can gate the whole pass".

The pattern

Entry 17 reasoned that the stale window's other user made a second pass possible. This entry is that reasoning being wrong about tense — it had already happened, one pass earlier, and left a 503 in the log as a receipt. When a guard is found to be measuring the wrong thing, check whether it has already failed before filing it as a risk. The evidence is usually in the same logs you are already reading.

19A limit with no ordering is a sample, and it decided what cars were worth

valueCar fetched comparables with one query: make, model, price not null, .limit(400). No ordering. For a model with more than 400 live listings that returns an arbitrary 400 of them — arbitrary in the strict sense, whatever the planner produced that day.

Measured 2026-08-14: 86 models hold more than 400 live listings, covering 112,927 of 158,540 — 71% of the corpus. A 2022 Outlander valuation was built from 9 of the 241 that exist, because nine happened to fall in the slice.

Why it went unnoticed for so long

Because until that morning it barely mattered. Every comparable was corrected onto the subject and combined into a weighted median, and a biased sample of a large set still lands near the middle of it. The error was real and small, and nothing pointed at it.

0.7.0 made it critical, and 0.7.0 was mine. Restricting comparables to the subject's own model year — a change that cut held-out error 19.3% and is right — turned a mild sampling bias into the whole basis of the estimate. The restriction is only ever as good as the sample of that year, and it was building answers from nine arbitrary cars while 232 sat unread. I shipped it without asking what fed it.

The tell

sanity-valuations.ts reported the worst case at 7.9% — inside its own 10% alert threshold, and easy to file as ordinary dispersion. It was not dispersion. Asking why that number was 7.9% rather than accepting it as tolerable is what found this; the check had been reporting the symptom for hours.

Fixed

The subject's model year is fetched on its own and in full, with neighbouring years fetched separately and bounded — their job is to measure the shape of depreciation, not to outvote the subject's year. Largest cohort in the corpus is 1,890 and only twelve exceed 1,000. Mean absolute gap across twelve common cars 2.57% → 1.98% on the mixed-basis yardstick then in use.

The pattern

Entries 9–12 were reads that failed loudly enough to be silent — a cap or a timeout swallowing rows. This is the same family with no error at all: the query succeeded, returned exactly what it was asked for, and what it was asked for was wrong.

A `limit` without an `order by` is a sampling decision, not a safety valve. Every one in this codebase should either be provably above the set it bounds, or be ordered so the rows that matter come first. And the corollary that cost the most here: a change that narrows what evidence is used makes every upstream sampling flaw newly load-bearing. Check what feeds a filter before you tighten it.

20The badge field holds whatever the dealer typed there

Trim is worth thousands and the valuation leans on it hard: an exact-badge comparable is weighted 1.5 against 0.8 for a mismatch, and once the subject's own trim has eight comparables everything else is dropped. That makes the badge field's cleanliness a price input, not a cosmetic concern.

Two failure modes, found 2026-08-14 while investigating why a 2023 Cupra Ateca VZX was missing from the year picker.

Fixed: a listing id inside the trim

cars4us publishes its badge as 110TSI Comfortline For Sale ID72363. The id is unique per car, so 857 of its 881 live listings had a badge nothing else could ever match — 857 distinct badges where 378 real ones exist. A trim that matches nothing is worse than no trim: those cars were weighted against their own identical twins and could never form a cohort.

Stripped at ingest and backfilled. The pattern is anchored at the end and requires the literal "for sale id" plus digits, verified to leave GLX (4WD) 5 Seat, Sale and For Sale alone.

The backfill was then partly undone, and that is the transferable part. A crawl pass was running with the old code, and when it re-reached cars4us two hours later it wrote 30 junk badges straight back — 378 distinct badges became

1. Nothing alerted; it was found by going to look. Fixing a value at ingest

and backfilling the history is only half a repair while a long-running process still holds the previous build: the process will undo the backfill, quietly, for as long as it runs. Either wait for the next pass, or re-run the backfill after it, and check rather than assume.

Open: badges that describe the whole car

The larger version has no such easy handle. Several sources publish the entire descriptor as the trim:

GSU50R GX 2WD White 8 Speed Sports Automatic Wagon
Series F30 LCI 320i M Sport Silver 8 Speed Sports Automatic Sedan
MQ MY16 Exceed Double Cab Blue 5 Speed Sports Automatic Utility

Every one of those is real information and none of them will ever match another car. Measured across 164,491 badged live listings:

Concentrated rather than spread: zaimotors carries 177 distinct badges across 178 listings, goulburnnissan 275 across 311.

Why this is not a quick fix. Unlike a listing id, these tokens sit beside genuine trim words and share a vocabulary with them. Sport is a descriptor in "M Sport" and a trim on a Ranger. Black Edition is a real trim; Black is a colour. normaliseBadge is already deliberately conservative for this reason — it refuses to reduce Sport GLS QF to GLS — and loosening it trades a visible duplicate for an invisible mismatch, which is the worse of the two.

Measured, and not worth doing

scripts/eval-badge-strip.ts puts it on held-out folds — 46,172 predictions across the 30 largest trim-varying models — against three strippers, each a superset of the last:

Noise. Not shipped.

Two reasons it does so little. The 5.3% is concentrated in a handful of sources rather than spread through the corpus, so it rarely touches the models with enough cars to be measured on. And badge levels already carry the load: a comparable of another trim is moved onto the subject's trim by its measured price difference, so exact string agreement matters much less than it looks like it should.

Worth keeping as a result rather than a plan. The risk of a greedy stripper is an invisible mismatch — merging a Black Edition with a black car — and paying that for 0.2% would be a bad trade made on a hunch. If a future extractor change makes badges dirtier, the harness is here to re-run.

21`last_seen_at` is not the index, and a clean zero was the tell

Some dealers retire a car without ever saying so. Westside Auto Wholesale answers a car that has gone with HTTP 200 and its generic listing page, titled Used Cars For Sale in Perth — a soft 404. Neither proof markDelistings accepts fires: not a 404, not a SOLD marker. The car stays live forever. On the 2026-08-15 pass, 302 sources reported 1,401 vanished cars that could be proved neither sold nor still listed, and 268 of 698 sources have never recorded a single sale, which is why so many dealers show no days-to-sell.

The proposed third proof was that our own extractor finds no car on the page. It is a dangerous one — a broken extractor returns null for every page and would invent a sale for every car on the site — so it needed measuring before anything else. Two hand-checked Westside pages had already shown the pattern exactly: 200, no SOLD text, extractor returns null.

The harness sampled two arms per source: live rows the newest complete traversal had seen, and live rows it had not, the second standing in for the delist check's candidates. It probed 144 pages across six sources and every single one read as a car. Zero nulls, in both arms.

That is not a finding, it is a bug, and the shape of it says so. A phenomenon observed by hand on two pages does not vanish across 144 without a reason, and a result that clean deserves less trust than a messy one.

The reason: candidates come from the index, not from `last_seen_at`. markDelistings takes presentUrls, the set the traversal actually saw, and the crawler is explicit that "a listing still present in the index but beyond this run's detail budget is protected by presentUrls". Such a car is emphatically still for sale — it just was not re-fetched, so last_seen_at stays old. The harness read that stale timestamp as vanished and filled its candidate arm with cars that were never candidates. They were on the site, in the index, and extracted perfectly. The zero was structural. It could not have come out otherwise.

The denormalised field looked equivalent and encoded a different question: last_seen_at answers "when did we last fetch this", the delist check asks "was it in the index". Those diverge by exactly the detail budget, which on a large source is most of the inventory.

So the harness was deleted rather than patched, along with the RPC that fed it. A wrong reconstruction of a production set is worse than none: it answers confidently in the right units. The measurement now runs inside the real delist check, where the candidate set is correct by construction — markDelistings counts how many refused candidates no longer describe a car and reports it on the run, and decides nothing on it.

The rule. Reconstruct a production set from the same input production uses, or do not reconstruct it — instrument production instead. And when a measurement disagrees with something already verified by hand, the measurement is the suspect.

22The seller said SOLD and we filed the car as stock

soldMarker reads a dealer's own statement off a page — a title retitled SOLD 2023 Mitsubishi Outlander ES ZM …, or the plain sentence this vehicle has been sold. It is the strongest evidence in the crawler, stronger than a 404, because a 404 is ambiguous and this is the seller stating the outcome. It is already accepted proof for a delisting.

It was called in exactly one place: the delist check's verify step, which only ever runs on a car that has left the index. A dealer whose index keeps the car was therefore never asked at all — 31 sources holding 2,035 cars have never produced a single delist candidate, which is what an index that drops nothing looks like from outside. Meanwhile the detail loop fetched those same pages every pass, handed them to the extractor, read the SOLD straight past, and stored the car as live stock.

Measured against retained captures, with no fetching — which is what retention is for — 1.2% of live listings on a uniform draw already carry a marker, roughly 2,200 cars, sitting in dealer stock counts, median asks, and valuation comparable sets. Per source the rate is 2.3%, and quoting that as a corpus figure would have overstated it by half: stratified sampling answers how many dealers are affected and weights a forty-car dealer like a nine-thousand-car one.

Two dealers show the shape. cheapcarstownsville answered 32 of 88 pages with a marker; 16 were cars we held live, 16 were cars we never had. BDK Automotive keeps sold cars listed permanently behind "This vehicle has been sold but we're happy to help you" under the generic title All Stock — 96 of 137.

Reading it on the detail path is not a new inference. It is the same proof, applied where nobody was looking. What needed care was everything around it.

The guard fired after the damage. The first version stopped the sweep when markers exceeded half the pages found — which happened on the 127th of 252, with 96 delistings already written. They were correct, but had the marker been broken they would have been 96 fabricated sales. A guard that stops after most of the damage is not a guard. Sold pages are now collected during the traversal and decided together once it ends, so nothing is written until the whole picture is known.

The guard measured the wrong thing. Counting pages that carry a marker punishes precisely the dealers this was built for: one who never removes a sold listing shows a permanently high marker rate forever, tripping on cars already delisted. The threshold now counts cars the run would newly retire against the cars the source holds live. A broken marker takes nearly the whole inventory at once; a sold-out dealer does not. The re-run of bdkauto recorded 0 and tripped nothing, which is what idempotent looks like.

A delisted car came back by itself. observeListing resurrected any delisted listing on sight, and these cars stay in the dealer's index — so each pass would have flapped the status and written a relisted event. Only the car's own page may relist it now. The index is the thing that lags: one Dealer Studio index served a stale price thirteen hours after the detail page changed, and 1,401 cars a pass sit missing from an index while still answering 200.

The rule. Evidence the crawler already trusts should be read everywhere the crawler already looks. And a safety threshold has to be denominated in the harm it prevents — cars wrongly retired — not in whatever is easiest to count at the time.

23Twelve of twelve: the cars Westside never said goodbye to

Follows on from entry 21, which left a question open. A soft 404 — a car's page answering 200 with the site's generic listing page and no SOLD text — defeats both proofs the delist check accepts. The rate was unmeasured, and the first attempt to measure it sampled the wrong population entirely.

It is measurable cheaply once the right population is asked. A car missing from the index for a few hours usually just fell out of a lagging index; a car missing for five days is a different claim. So: Westside's twelve longest-vanished live listings, fetched directly. Twelve polite requests, not the 2h43m a full traversal of that source costs.

All twelve last seen 10 August. All twelve titled *Used Cars For Sale in Perth

• Westside Auto Wholesale* rather than a car.

Read against the same source's delist check, which asked sixty candidates and found zero soft 404s, the two results are not in tension — they are the same finding from both ends. The check's sixty were the recently vanished, held there by the id ordering of entry 21, and those are genuinely still for sale: an index drops a car briefly and picks it back up. The cars gone five days are really gone. Which candidates you ask decides what you learn, and the old ordering could only ever ask the ones that would say "still here".

This is why the rotation had to land before the signal could be judged, and why last_seen_at is the tiebreak among never-checked candidates rather than id.

Still not acted on. Twelve for twelve is a rate on one dealer, and the signal's failure mode is not subtle: a broken extractor returns null for every page and would invent a sale for every car on the site. The run now counts these beside the parser's own health on detail pages, measured separately from listings_extracted because that counter includes inline listings and would read ~100% even with detail parsing dead. When the count and the health are both in hand across many sources, the question can be decided on evidence rather than on one dealer in Perth.

24A field at exactly 100% empty is a question, not an answer

Two extractor defects turned up on the same day from the same question — why is this field empty? — and both had the same shape: a guard correctly refusing bad input, with nothing behind it.

cox-radius promoted a field only when the page yielded exactly one candidate value, which is right, and fired on every page because a radius detail page always carries "recently viewed" cards for other cars. 1,437 live listings, no price on 1,404. carsguide's extractor serves two sites, and on autotrader the spec table flattens differently and delivers the body-type cell as navigation text — s Convertibles Dual cab utes Hatchbacks People movers Statio — which shortSpec refused. 19,221 listings, no body type on all of them. Both were fixed by giving the guard a second source it could trust: the page's own JSON-LD in the first case, the URL segment the site always publishes in the second.

So the method, once it was clear this was a class rather than two accidents — null rates per field per platform, with nextjs-embedded as the control at 0.5–2.3% across everything, which is what a healthy extractor looks like:

select s.platform, count(*) as live,
  round(100.0*count(*) filter (where c.drivetrain is null)/count(*),1) as drive_pct
from listing_current c join source s on s.id = c.source_id
where c.status = 'live' group by s.platform having count(*) > 500;

Exactly 100.0% is the signature. A field the site publishes and the parser sometimes misses lands somewhere untidy; a field no code ever reads lands on a round number. That is how both defects announced themselves.

And it is only a signature, not a verdict. The same sweep flagged easycars at 100.0% for drivetrain, and its extractor has no drivetrain logic at all — so the signature was perfect. But the pages carry no drivetrain text either, in long or short form, not even on a 2013 Audi Q7 that is certainly quattro. The platform does not publish it, 100% empty is the correct answer, and there is nothing to fix. Checking cost three retained captures and no fetches.

Worth re-running after any extractor change, and worth remembering that the prize is a field the source publishes — not a field we would like to have.

25The corpus could not accept an improvement

Every distinct body is retained so that a better parser can re-read the past without re-crawling. That was half a mechanism. The other half — the write path being willing to take the improvement — did not exist, and nobody had noticed because until 2026-08-15 no extractor fix had ever been re-parsed at scale.

Three extractor fixes landed that day. None of them could have reached a single car already in the corpus. They would have improved only vehicles inserted afterwards, which for autotrader's 19,221 bodyless listings means never. The failure was three layers deep and each layer hid the one beneath it.

One: a sentinel is not a null. condition is written on insert as l.condition ?? "unknown". The listing row already has a fill-in block for exactly this problem — its comment reads "a listing first seen by a weaker parser kept its gaps forever" — and condition was missing from it, because the rule is blanks only and "unknown" is not blank. It is a blank wearing a value's clothes. Re-parsing recovered every price and no conditions at all.

Two: vehicle attributes were written once. Fixed the sentinel, re-parsed again, recovered every condition and no body types. Body type, transmission, drivetrain and fuel live on vehicle, and resolveVehicle finds a row by fingerprint and returns it untouched. Everything there was written at first sight and never looked at again.

Three: the fill was on a path a re-observation never takes. Fixed that, and still nothing, because resolveVehicle is only called when a listing is created. A re-observation of a known listing never reaches it. The fill had to move to the branch that actually runs.

Then it worked: 6,032 listings across nine sources recovered fields, with no page fetched. pattersoncheney went from 0 body types to 1,154 and 9 prices to 599; five BMW dealers gained 2,311 drivetrains.

reparse.ts needed its own three corrections first, all of which would have corrupted data at scale. It took the newest N captures, so it re-read one busy car five times and never reached the quiet one. It was unpaged, so PostgREST's 1,000-row cap truncated it while reporting success. And it wrote inline listings with the default source view of detail, mislabelling a search-index price as a listing-page price — the confusion that once manufactured six price changes no dealer made, and one that check_cross_view_price_changes cannot catch, because the mislabel removes the boundary the check looks for.

It also had to learn not to raise the dead. observeListing relists a delisted car when a detail view shows it again, correctly. Feeding it a capture from last Tuesday would resurrect every car sold since — bdkauto alone had 96 confirmed sold whose pages still said so. Detail re-parses are filtered to live listings.

The rule. Retaining the evidence is worth nothing if the corpus cannot accept a better reading of it. Any field written on insert and never revisited is frozen for the life of the row, and freezing is silent: the extractor improves, the tests pass, the numbers do not move, and nobody asks why. When adding a field to a row that outlives its first observation, decide then whether a later, better reading may fill it — and if the answer is yes, write the fill at the same time.

26One car, thirteen listings, and a definition that only travelled halfway

65% of vehicles now carry a VIN, which means the corpus can tell that two listings at two yards are one physical car. It already knew: 34,431 cars are visible at more than one source, one of them at sixteen. Of 184,739 live listings, only 119,990 are distinct cars — 35% are the same car advertised more than once, and where a car is priced at two yards the asks agree 97.6% of the time. These are syndication copies, not competing offers.

That fact has one consequence above all others: a car vanishing from one index is not a sale. 2,376 of the 9,841 delistings in 30 days with a known VIN — 24% — are cars still openly for sale somewhere else we watch.

The project had already learned this. Migration 0135 gave the pulse and the weekly report one definition of sold: one row per vehicle, and only once no watched listing of that car is still live. It was written because the /reports chrome said $181.4M for a week the report below it called $88.6M. Migration 0144 gave the market boards the same rule.

Two surfaces never got it, and the reason they were missed is the useful part: the guard is a correlated subquery, and PostgREST cannot express one. Every surface that could be fixed in SQL was fixed. Every surface reading the table through the client kept the old meaning, because there was no way to write the new one without moving the query. The fix was structural and the drift followed the query language, not the code review.

• The home page sale tape read listing_current filtered to delisted. Its own doc comment called the rows "the latest cars confirmed sold". Eight of the fourteen it was serving were cars still listed at another yard, several at the same price.
• Valuation's days-to-turn had neither rule, so one car syndicated to three sites and dropped from one counted as a sale, and dropped from all three counted as three.

Days-to-turn is the better warning. Globally the error is invisible — the median advertised age is 34 days with the guard or without it, because the false exits are distributed like the real ones. A single aggregate check would have passed. Per model, which is the only way the valuation ever asks:

Both directions, and large enough to invert the advice. A bias that cancels in the aggregate is not a small bias; it is a bias you cannot see at the altitude you are checking from. Check a derived number at the grain it is consumed at, not the grain that is convenient to query.

Moving days-to-turn into an RPC lifted a second, silent fault on the way past: the client read was capped at PostgREST's 1,000 rows, so common makes' cohorts had been truncated with no error and nothing to show it. See [21] — the same shape as a clean zero.

27A third of the corpus was read by the fallback parser, and nothing was wrong

Sources are seeded with a platform, and one that nobody detected gets jsonld. That is a sensible fallback: almost every dealer site emits a schema.org Vehicle node, so the source works immediately. It just works badly. JSON-LD carries make, model, year and price and not much else, and store.ts deliberately prunes the whole blob afterwards because those four values are already in typed columns — so a misfiled source stores four vehicle fields and an empty `fields` map, while the same page's own stock JSON sits in the retained capture carrying VIN, odometer, and the seller's listing age.

Measured across all 495 enabled jsonld sources: 440 had a better extractor already sitting in the registry, covering roughly 63,000 live listings — about a third of the corpus. 338 were Dealer Studio sites, 102 were i-Motor.

Trivett is the clearest case. As jsonld: make, model, year, price. As imotor, on the identical retained bytes: 19 vehicle fields, VIN, odometer, and a createdAt that advertisedDays already knew how to age. 1,202 live listings, and nothing had to be fetched to get any of it.

Why it stayed invisible. Every failure mode this project has learned to watch for was absent. No errors. No zero-yield sources — they all returned listings. No stale timestamps, no failing integrity check, no gap in the freshness numbers. The sources looked healthy on every axis being measured, because they were healthy on every axis being measured. The only symptom was poverty of fields, and nothing was counting fields.

That is the general shape worth remembering: a fallback that works is more dangerous than one that fails. A fallback that fails gets fixed the first week. A fallback that quietly returns less than it could is indistinguishable from a thin source, and "that dealer doesn't publish much" is an explanation that never runs out. It was only found by asking a different question — not "is this source working" but "is this the best reader for this page".

What made it safe to act on. More fields is not the same as better data, and the way this could have done real damage is by changing identity: reparse and the crawler both key on URL, so an extractor deriving a different URL would have created duplicates rather than enriched rows. So the tool refuses to move a source unless the challenger returns a listing at the same URL on every sampled page. Field count decides whether to look; identity decides whether to act. Across 440 sources it refused none.

Verifying 88 pages over 30 sources before applying: URL identical on all 88, no nulls, one price recovered from undefined, and eight make/model disagreements — every one of them the new extractor being more complete (land rover range → land rover range rover evoque, chery tiggo → chery tiggo 9). Not one disagreement favoured the incumbent.

The sweep now runs weekly beside discovery rather than in the crawl chain, because seeding is when the mistake is made. See scripts/refit-platforms.ts and deploy/run-discover.sh.

28"The newest one we can read" is not "the newest one"

While re-parsing 149 sources onto their corrected extractors, the corpus recorded 26 price changes that no dealer made. A Lexus NX at Trivett went 39,990 to 37,990 on 14 August — a real cut. On 15 August a re-parse read the 13 August capture and filed a 2,000 rise straight back to 39,990.

The cause is one line in reparse.ts, and specifically the order of two conditions in it:

.not("body_key", "is", null)     -- filter, in the query
...
if (seen.has(c.url)) continue;   -- then dedupe, in the loop

Read as "the newest capture of each URL". It means "the newest capture of each URL that still has a body", and those are different rows far more often than the code's author could have known: bodies are sampled rather than kept for every fetch, and 42% of the last week's captures have no body at all. So for a large share of listings the newest readable capture is a day or two old, and re-parsing it writes that day-old price forward as the newest observation. The diff against the real price becomes an event.

A sampling policy changed the meaning of a query in a tool that predated it. Nothing in reparse.ts was edited when body retention became selective; its query kept returning rows, and the rows kept looking like captures. This is the same failure as DATA-QUALITY 1 — diffing two readings that are not comparable — arriving by a route the original fix did not cover.

It hid because it needs two things at once: a capture whose body was dropped, and a price that moved before the next retained one. Prices mostly sit still, so 30,000 re-parsed listings produced 26 fabrications — 0.2%, which is small enough to look like ordinary market noise on any dashboard.

How it was actually caught. Not by a check. The corpus monitor happened to report crawling=24 while the re-parse was running, and a crawl writing fresh prices into the same listings a re-parse is writing old ones into is worth thinking about before it is worth measuring. The question "could a stale write manufacture a change?" came first; the query confirming it came second. No alert would have fired: every check passed throughout.

The fix is to swap the order — dedupe by URL across every capture, then drop the URLs whose newest capture is one we cannot read. A skipped listing keeps the data it has and gets re-read by the next crawl with the current extractor, costing hours. Trivett skips 8 of 1,202 that way, 0.7%.

The cleanup was 26 price_change events, matched on listing, timestamp and both price values, and deleted. The stale observations stay: an observation is a true record of what a capture said, and the corpus is append-only. They make listing_current show a wrong price for those 26 cars until the next pass overwrites it, which is a few hours of a wrong asking price against a permanent fabricated event — the right way round.

check_observation_capture_order now looks for it. A warning, not a failure, because a crawler writing while the check runs can produce a benign near-tie, and a check that cries wolf gets ignored — which is its own DATA-QUALITY entry waiting to happen.

The general form, worth carrying: when a storage policy becomes selective, every query that reads that storage acquires a new meaning, and none of them will mention it. Retention was a change to how much disk the project used. It was also, silently, a change to what "latest" meant.

29Two rows for one car, and the check that could never say so

check_duplicate_vins has returned 0 every time it has ever run, and always would have. It counts vehicle rows sharing a VIN; vehicle_vin_key is a unique index. The check asserts the thing the database already guarantees.

The reason it exists, written beside it in check-integrity.ts, is exactly right: "two vehicles sharing one means the identity resolution is broken and comparables are double-counted." It was looking for that in the one shape it cannot take. 2,100 cars were stored as two vehicle rows each — one holding the VIN, one holding no VIN at all — and nothing in the suite could see them.

How the second row appears

resolveVehicle looks a car up by VIN first and by a source-scoped src:<source>:<id> fingerprint second, and it runs only when a listing is created. Re-observing an existing listing never re-resolves it. So a listing first read by a parser that could not see a VIN gets a fingerprinted row and keeps it for ever, however much the parser improves; the same car on a sister site, read by a parser that could see one, gets a second row keyed by VIN.

Both rows are honest records of what was read. Together they are two cars where there is one.

The corpus was detecting this and throwing the detection away. fillVehicleBlanks recovers the VIN on a later pass, tries to write it into the blank row, and the unique index refuses because another row holds it. That refusal is not a failure. It is a positive identification by the one identifier this project trusts, and it went to a console.warn in a log nobody keeps.

Found from the far end

Toowoomba GMSV and Toowoomba Automotive are one yard behind two domains, on the same platform, serving identical detail paths — /view/2014-Hyundai-i30-U19524-KMHD351EMEU183127/34673102 — down to the same trailing stock id. GMSV was read as cox-radius and Automotive as jsonld, and only the second could read a VIN. Of GMSV's 78 cars, 64 collided with a row Automotive already held; 14, the ones Automotive did not carry, took their VIN cleanly. The pair's live listings covered 142 vehicle rows for 78 physical cars.

The sweep that followed found the same shape at 25 other dealers, all of them groups running a site per marque: buckby (226 + 221 + 142), village (140 + 139 + 114), frankston, llewellyn, tynan, taree. This is not one misfiled dealer. It is what a dealer group looks like when half its sites can read a VIN and half cannot.

What a duplicate costs

A sale that never happened. The sold guard that 0135, 0148 and 0149 all share is no live listing shares this vehicle_id. A second row hides the sister listing from it. Three Toowoomba cars delisted from both sites within an hour of each other on 2026-08-15 — 07:44 from Automotive, 08:47 from GMSV — and were recorded as six sales. This is entry 26's problem arriving underneath its own fix: 26 moved every surface onto one definition of sold, and that definition is keyed on vehicle_id.

A car voting twice on its own price. valuation.ts dedupes comparables by vehicle_id and says why: "Safe to merge because vehicle identity here is VIN-based." True in the direction it was worried about — a fingerprinted car is never wrongly merged — and silent about the other one. Measured on the shipped model:

And often at two different asks — the same Atto 1 at $23,888 on one site and $23,990 on the other — so the duplicate does not merely double a vote, it widens the range while doing it.

Cohorts that are smaller than they look. Counting cars rather than rows: GMC Yukon 2024 4 → 3, Denza B5 2026 5 → 4, BYD Atto 1 2025 11 → 9, BYD Sealion 5 2025 17 → 14. Corpus-wide the error is 1.6% of vehicle rows and invisible; per cohort, which is the only grain a valuation ever asks at, it reaches 25%. Entry 26 made this point about days-to-turn and it is the same point: a bias that cancels in the aggregate is not a small bias; it is a bias you cannot see at the altitude you are checking from.

The repair, and why it is not a source merge

0082 already drew this line. One page filed under two source ids is a duplicate and gets retired; one car on a marque site and its group site is two real pages, two real listings, and "stays counted twice on purpose — valuations dedupe by vehicle where it matters." Nothing about the source or the listing is wrong here. The vehicle layer is.

So listing.vehicle_id is repointed at the row holding the VIN, and the emptied row is retired naming its survivor (vehicle.merged_into_vehicle_id, 0154). Repointing the column rather than resolving a pointer at read time is the whole design: every reader here already asks about a car by vehicle_id — count(distinct vehicle_id) in the pulse, the boards and the ladder, bestPerVehicle in the valuation, the sold guard in forty-odd SQL functions — so they are all correct with no change at all. A read-time pointer would need every one of them to learn about it, and the one that forgot would double-count in silence. That is 0082's own argument for a status value over a flag, pointing the same way one grain down.

The VIN holder survives because resolveVehicle matches on VIN first: retire that one instead and the car re-splits on the next crawl.

Nothing is deleted, no observation is touched, and no listing changes status — a merge cannot manufacture a delisting or resurrect one. merged_from_vehicle_id records where each listing came from, so the whole sweep is reversible.

A sweep was not enough, and the measurement said so

The first design left the crawl recording collisions and a periodic sweep acting on them. Then the sweep was re-run ninety minutes after clearing 2,006 rows, and found 289 fresh duplicates across four dealers that had not been in the first pass at all.

The reason is structural and was in plain sight: a listing is usually created from an index reading, and an index carries no VIN. resolveVehicle runs at creation, gets no VIN, mints a src: row — and only when the detail page is read does the corpus learn which car it is, by which time the listing exists and nothing re-resolves it. Every listing first seen from an index is a candidate duplicate. A weekly job against a source refilling that fast is not a fix; it is a job that guarantees the check it exists to satisfy never reads zero.

So the collision now acts where it is detected. fillVehicleBlanks moves the listings and retires the row on the spot, under the same agreement guard, and returns the surviving id so a listing being created lands on the right car. The sweep stays for the backfill and for the URL-slug route the crawl path cannot see.

Worth being precise about why this is not the taxonomy exception the same function stops short of. That one is heuristic — string-matching a model name — and is rightly deliberate and reviewable. A VIN match is not a guess. It is exactly what resolveVehicle would have done with the same reading had the VIN been legible when the listing was created; the correction is that act arriving one pass late, not a new judgement.

Identity decides whether to act. A VIN counts only where a vehicle row already holds it, so no identity is invented from a URL slug; and two rows are only merged where they agree on make, model and year. That refuses 88 pairs, almost all of them one parser reading a shorter name — "Toyota Landcruiser" against "Toyota Landcruiser Prado", "GWM Cannon" against "GWM Cannon Alpha", "SsangYong Rexton" against "KGM Rexton". They are one car and merging them here would still be wrong: the survivor is whichever row holds the VIN, so the merge would move those listings onto whatever name that row happens to carry, and make, model and badge are what decide the cohort a car is valued against. Silently moving a Prado into the Landcruiser cohort is worse than leaving it counted twice. They are taxonomy work, and they get their own count (vin_pairs_disagreeing_on_identity) so the merge count can be driven to zero.

Two things the fix got wrong first

The new check read a column, not a payload. It recovered the VIN from listing_observation.fields on the newest observation — which is what that column was for when it was written, and has not been since 0024 made storage hash-deduped. 983,830 of 1,325,603 present observations, 74%, store `{}` and a hash, and listing_current resolves them back with a lateral join that the check did not have. It reported 1,519 duplicates where the sweep found 2,100; through the resolved payload, 2,092. Sampled directly: of 120 listings whose resolved payload carries a VIN, 71 have it only after resolution.

This is entry 28's lesson recurring in a query written the day after 28 was written down. When a storage policy becomes selective, every query that reads that storage acquires a new meaning, and none of them will mention it. Knowing the rule is not the same as applying it, and the tell is the same both times: a number that disagrees with another measurement of the same thing.

A retired row still answers to its fingerprint. The sweep cannot move the fingerprint onto the survivor, because that fingerprint is what the source will look the car up by on its very next pass. So resolveVehicle would have found the retired row, hung a new listing off it, and re-split the car within hours — with check_retired_vehicles_own_no_listings failing and nothing saying why. Caught by watching the crawler write while the merge was running, which is the same way entry 28 was caught. Resolution now follows the retirement pointer; it is the one place in the codebase that has to know about it.

30The model field holds the dealer's filing system

Entry 20 found a listing id inside the badge at cars4us and stripped it. The same disease was in the model field at another source the whole time, and the search that found the first one could not see the second: it looked for a literal "For Sale ID", which is one CMS's way of saying it.

dealer:toowoombagmsv runs on Cox's radius platform, whose detail URLs are /view/<Year>-<Make>-<Model>/<id> — except this dealer appends its own listing reference:

/view/2014-Hyundai-i30-U19524-KMHD351EMEU183127/34673102
             |    |   |      |
             |    |   |      +-- VIN
             |    |   +--------- stock number
             |    +------------- model
             +------------------ make

The extractor took the year, matched the make against the canonical list, and called everything left the model. So the model was i30 U19524 KMHD351EMEU183127.

Measured 2026-08-16: 60 of that source's 132 live listings, and 66 vehicle rows corpus-wide. Every one of the 85 taxonomy_alias rows the source has recorded carries a VIN in raw_model — an alias per car, which is an alias that can never match a second listing.

Why this is worse than a wrong value. A stock number is unique per car and a VIN is unique per car in the world, so each of these models was a cohort of one. The car does not appear on /model under a wrong name; it does not appear at all, and it can never be a valuation comparable for anything, including its own identical twin. Nothing looks broken from the outside — a Kona is simply missing from the Kona page, and no count anywhere goes red. This is entry 20's "a trim that matches nothing is worse than no trim", one field over and with the volume knob turned up, because the model is the cohort.

It had been met before and treated one car at a time. Migration 0049 carried four hand-written aliases — ['BYD', 'atto 3 u19667 lgxce4cb3p2182071', 'Atto 3'] and three like it — each naming one car's mangled model as a synonym for the real one. That is the correct repair for a typo and no repair at all for this: the reference is unique per car, so every car that arrives afterwards is a fresh cohort of one needing a fresh alias. Sixty had accumulated in the five weeks since. An alias list is the wrong shape for a defect that is generative — if the fix has to name each instance, the instances will outrun it.

The fix strips the tail structurally: a trailing VIN — seventeen characters of the VIN alphabet, no I/O/Q, letters and digits both — and then, only if a VIN was found in front of it, a trailing stock number of one to four letters and at least four digits. Both tokens or neither, and never the last token standing.

That the VIN is required is the whole guard, and it was measured rather than assumed. Twelve stored models end in something stock-shaped with no VIN behind it, and all twelve are Avida motorhomes whose real model codes are Bruny B7042 and Explorer LX V5912. A rule that stripped a stock-shaped tail on its own would have eaten them. Nothing in make or badge matches the pair at all, and no model ends in a bare VIN.

The rule lives in stripStockAndVin in taxonomy.ts, beside the cars4us one it is a sibling of, so it applies to every source at ingest and not only to the one that was caught. The extractor applies the same test to the slug's tokens before they are joined, which is how the VIN survives as a value.

The VIN was kept, not discarded. Checked before it was trusted: across 333 retained captures the slug's VIN matched the page's own vehicleIdentificationNumber 333 times and disagreed none, and the stock number matched the page's sku on all 333 likewise.

And keeping it immediately found something else. Of the 78 distinct vehicles behind those 132 live listings, 14 took their VIN and 64 were refused by `vehicle_vin_key` because another row already held it — all 64 of them held by `dealer:toowoombaautomotive`, a sister site on a different domain and a different extractor, which has been reading VINs all along. So 64 cars were in the corpus twice under two source ids, and it was invisible until one side of the pair started reading the identifier that proves it.

That is entry 29's defect, and this is worth recording as an instance of it rather than a separate finding: a duplicate vehicle is not discovered by looking for duplicates. Nothing about either row was wrong, no count went red, and no string comparison would have matched them — they were two different dealers' spellings of two different pages. What surfaced all 64 in one pass was an extractor learning to read an identifier it had been throwing away, and the unique index then refusing the write. The check that could never say so, in entry 29's title, could not say so because the fact it needed had not been extracted yet.

Resolved as of 2026-08-17: fillVehicleBlanks no longer merely records the collision, it adopts the listing into the VIN-holding row (entry 29), and all 64 pairs have collapsed. Those 78 vehicles now carry a VIN apiece, 64 of them holding listings from both sites, and no VIN in the corpus sits on two rows.

Re-parsing does not repair this, and that is by design. fillVehicleBlanks fills blanks only and excludes make, model, badge and year, because those decide which cohort a car is valued against and moving one silently would change an answer already given. So the extractor fix reaches new cars only; the 66 stored rows are repaired by recanonicalise.ts, where it is deliberate and reviewable, filtered server-side and looped until the filter comes back empty so PostgREST's 1,000-row cap cannot quietly cover a fraction of it.

Entry 20's warning applies in a narrower form. A crawl pass running the old build cannot re-mangle a repaired row — the model is written once, at vehicle creation — but any new listing it creates before this ships arrives mangled. The sweep is idempotent and re-running it is the cleanup. Re-measured 2026-08-17 after a night of crawling on the old build: still zero, because the source minted no new vehicle rows. The hazard is real and has not fired.

One ordering worth knowing. stripStockAndVin runs inside tidyModelString, which resolveVehicle applies to the model and badge it is about to insert — after splitTruncatedModel and foldModelIntoBadge have already read the raw string. For cox-radius that is moot, because the extractor removes the reference before store.ts ever sees it. It matters only if some future source glues a reference into a model that also needs folding: a Mercedes "C63 U19524 KMH…" folds to model "C-class" with the reference in the badge, where tidyModelString still strips it, but a BMW "X1 U19524 KMH…" leaves a two-token badge that the three-token guard declines to touch. Not fixed here, because no source does this and loosening that guard to cover a case nobody exhibits is how a narrow rule stops being narrow.

The general form: a unique-per-car identifier in a grouping field does not corrupt the group, it deletes the row from it. Wrong values are visible because they sit next to right ones. A value that can never match anything sits next to nothing at all, and every count of what is present stays correct.

31The suite was blind to the subsystem it gated

A proposed fix — fitting yearPriceLevels within trim, or on trim-residualised prices — was refuted by an adversarial review that did the one thing the proposal had not: it built the patched engine and measured it.

It improves zero of six target cohorts and regresses four. Volkswagen Passat 2023 goes from 0.0% to -20.2%, Honda CR-V 2026 from -2.4% to -15.2%. The mechanism is that trim and year are collinear in exactly the cohorts that reach the level path: Passat 2022 stock is 23 of 31 one trim, 2023 is six trims, and a median trim offset cannot separate "2023 stock is dearer trims" from "2023 is dearer" — it assigns the whole effect to trim and strips the real year signal.

But the finding that matters is the one about the harness.

`restrictToYear` fires for all twelve sanity cases and all seven anchors. Inside a single model year the subject's level and the comparable's are the same number, so the year term is identically zero, and the patched engine left every one of them byte-identical — while being live on 36.6% of graded cohorts. "The sanity twelve are unmoved" would have been reported as evidence of safety. It is evidence of non-coverage wearing the same clothes.

This is DATA-QUALITY 26 again — a bias that cancels in the aggregate is one you cannot see at the altitude you are checking from — and it was walked into three days later, by the author of that entry, in a different dimension. The generalisation worth keeping: a regression suite proves nothing about code paths its fixtures cannot reach, and a suite that cannot reach them will report that fact as success. Before trusting an unmoved suite, ask which branch the change lives on and whether any fixture takes it.

sanity-valuations.ts now carries a second section that does, asserting properties rather than gaps — because the cohorts that reach the year path are thin in the subject's year, which is the same thing as having too few asks to be a yardstick. Its load-bearing assertion is that some cohort reaches the path at all: if every candidate drifts into restrictToYear as the corpus grows, coverage returns to zero silently, so zero exercised cohorts is itself a failure.

32Two hosts, one table, and an alert that cleared itself

A genuine integrity failure — 17 checks, one failing — was masked four minutes later by a passing run of 16 checks from the crawl box, which sat on a commit predating the seventeenth check. health.ts read the most recent integrity_run row, so the alert cleared and nothing had been fixed.

That is the worst behaviour available to a sentinel, and it was found by reading the row rather than by trusting the alert.

Two things were wrong at once. The signal did not distinguish the production run from a developer running the suite by hand, so whichever host ran last spoke for the corpus. And version skew was invisible: a scheduled run from an older commit can pass a suite that no longer exists, and nothing said so.

Runs now record scheduled (the chain exports SPOTLOT_CHAIN=1) and their check count. health.ts reads the latest scheduled run, and says out loud when production is running fewer checks than some run has carried. A developer still gets their non-zero exit; they can no longer silence production, nor alarm it.

The class: when a signal moves from one writer to two, "the latest row" stops answering the question it used to answer. The same applies to integrity_run, to migration numbering, and to any table where a laptop and a server both have credentials.

33Ten thousand fetches a day, at sites that told us no

market:autotrader fetched 1,500 index pages on each of six fast passes in one day and found zero listings every time. market:carsguide did the same for 1,082. Ten thousand requests a day returning nothing, at two sites that already restrict automated access. The deep pass on the same source finds 26,665, so the sites are readable — just not by an index-only pass.

It cost more than politeness. Autotrader was the fast pass's last fifty minutes, producing no observations, so lastObs aged past the 45-minute stall threshold while the pass was running correctly. The alert fired twice, was accurate about its symptom and useless about its cause, and both times the honest reading was "this is fine" — which is how an alert teaches its reader to stop looking.

Marketplaces are now deep-only on the schedule; a source named explicitly by id is still crawled either way. The fast pass went from 63 minutes to 9½.

Found by chasing an alert that was, in itself, a false positive by sixty seconds — health sampled at 06:22, the pass closed its run row at 06:23:15. The alert was wrong and the thing underneath it was real, which is the argument for chasing them rather than tuning them out.

34A prefix is not a truncation

102 pairs where one VIN identified two vehicle rows that disagreed about the car got filed as "probably mostly dealer typos". Reading them says otherwise: in every case one model is a strict prefix of the other. Pajero vs Pajero Sport (27), Landcruiser vs Landcruiser Prado (22), Cannon vs Cannon Alpha (22), RAV vs RAV 4 Hybrid, Camry vs Camry Hybrid, Yaris vs Yaris Cross.

The temptation is a rule: same VIN, one model a prefix of the other, take the longer. It would have been a disaster. The prefix relation does not distinguish a truncated string from a real model that happens to be shorter. Landcruiser has 1,464 rows and Prado is its own model with its own market; Camry has 577 and Camry Hybrid is a different car. A prefix rule moves 22 cars into a cohort of 1,464 and wrecks both.

The distinguishing evidence is the badge. Toyota "RAV" carries badges that begin "4 Hybrid GX-2WD"; Toyota "RAV 4 Hybrid" carries "GX-2WD". One string, "RAV 4 Hybrid GX-2WD", cut at two different points by two parsers, and no cohort called RAV or RAV 4 Hybrid should exist beside RAV4's 1,726. Where the badge completes the model, it is a truncation and mechanically fixable. Where the badge is a trim and the short model stands on its own, it is one car mislabelled by one source and needs per-car evidence no rule can supply.

51 fixed, ~90 left as a warning — which is what the check that found them already called them: "taxonomy work, not a merge."

35Two sweeps an hour apart are not an A/B

A change to comparable dedupe was measured the obvious way: grade all 3,074 cohorts, change the code, grade them again. The second run came back 56 flagged → 71, with wild up 10→16 and overconfident up 14→25. Read straight, that refutes the change, and it was nearly reported as such.

It measures nothing. Two things were wrong with it at once, and either alone would have been enough.

The sweep cannot see this change. Its n_live, n_used and yardstickN are taken upstream of the dedupe, so a change that provably collapses Suzuki Grand Vitara 2017 from eight comparables to four — verified directly, four of its eight cars are one car on four shopfronts — moved not one of 3,074 counters. Zero cohorts lost a comparable. The instrument had no needle for the quantity being changed.

And its noise floor is larger than any effect. The runs were an hour apart on a live corpus that a crawl pass ran through, and 1,357 of 3,074 estimates moved between them. The regression list gives it away on sight: Ford Mustang 1967 at +23.1% → +92.3% on n 3->3, Lexus RC 2016 on n 1->1. Unchanged comp sets cannot swing sixty points from a dedupe that never fired. That was the market moving, plus tiny cohorts being tiny.

This is DATA-QUALITY 26 and 31 for the third time in a week — an instrument reporting confidently on something it cannot resolve. What is new is that here the wrong answer was decisive-looking: fifteen more flagged cohorts, in the exact categories predicted to improve, is the shape of a clean refutation. Committing the prediction beforehand ("what would refute this is wild rising") is what made it feel conclusive, and a stated prediction is worth keeping — but it only binds if the measurement can move for the reason claimed. Check that the instrument responds to the change before trusting what it says about it: had the counters been read for a single cohort first, all zero of them, the comparison would have been abandoned in a minute.

The replacement is scripts/eval-dedupe.ts: both variants in one process against identical rows, so there is no corpus to drift, held out by fold. It carries its own trap, which is the reason the answer is not obvious. Copies leak the answer. Leave a held-out car's duplicates in the training set and they carry its exact price — the model scores brilliantly by reading the subject off its own copy, and duplication measures as an improvement. In production the subject is a car someone asks about, not a row in the corpus, and no copy of it exists. So the subject and every fingerprint sibling come out of training before either variant runs.

Outcome. The change was reverted. Across 73,656 held-out subjects the fold made median error worse by 0.35%, and worse in every bucket with a real sample — +0.69% where it removed under 2% of the comparable set, +0.22% at 2-5%, +1.32% at 5-10%. 37,894 subjects degraded against 32,519 improved. The single improving bucket, 10-20% removed, holds 124 subjects and points the way the hypothesis predicted, which is exactly the result not to reach for.

The intuition it kills is worth keeping. Copies do over-weight whichever dealer runs the most shopfronts — that part was right. But a copy carries the same price, so it piles up at the median rather than dragging it, while removing it costs real sample. The variance that adds exceeds the bias it removes, and no amount of care about the merge rule's false-positive rate (2 in 11,526, itself half VIN error) could have rescued a change whose premise was wrong. Getting the safety of an operation right is not evidence that the operation is worth doing — the whole VIN-ground-truth measurement answered "can this be done safely?" and never once asked "does it help?"

Kept: scripts/eval-dedupe.ts, and 0172's has_vin on listing_current, which it needs.

36The confidence label is calibrated; the tail is not a bug

Twenty-three cohorts are flagged by two independent sweeps — the ones that survive both runs, as against the twenty-four that appear in only one and are drift. Thirteen of the twenty-three say high confidence while sitting 15 to 30% from their own median ask, all of them older cars in cohorts of eight to fifteen. Toyota Corolla 2008 reads 29.9% low and calls it high.

That looks exactly like a calibration bug, and the next step looked obvious: make high harder to earn on an old car. Measured first, across 2,965 cohorts, it is not a bug at all.

confidence        n   median |gap|   p90    over 15%
high           2,114        2.0%     8.3%      2.5%
medium           192        2.7%    15.1%     10.4%
low              626        5.2%    21.3%     20.0%
insufficient      33        7.0%    28.6%     42.4%

Monotonic in every column, across four levels. The label means what it says. Restricted to cars 2015 and older it stays monotonic and shifts: high runs 3.8% median and 7.4% past fifteen, against 2.5% overall. So thirteen bad high-confidence cohorts is not thirteen broken ones — it is 29 of 391, which is the 7.4% tail, showing up where a tail shows up.

The honest reading is that high is mildly optimistic with age, not wrong, and the fix that suggested itself would have been a change to a working model justified by its own tail. Worth keeping as the number to beat: high confidence is ~2% median error on modern cars, ~4% on pre-2015 ones.

The method generalises past this: a persistent-fault list needs two runs. One sweep flags 27 or 43 cohorts depending on the hour; the intersection is 23, and nearly half of any single run's flags are the corpus moving underneath it (DATA-QUALITY 35).

37The missing odometers are not missing

11.5% of live listings carry no odometer_km, and the metadata makes that look like parser loss worth recovering: of 3,000 sampled, 2,137 hold km, odometer_reading and odometer_unit in their fields blob, and 365 more hold odometer. Fields present, column null — an obvious backfill.

Reading the values ends it. 12,714 are the literal string `0`, and exactly two are a number a car could have travelled. Dealers publish km: 0 on undriven stock and the parser is right to refuse it: zero is the absence of a reading, not a reading of zero.

The condition split confirms it from the other side — 29.8% of new listings have no odometer against 3.3% of used, which is the only population that becomes a comparable, and those already enter at half weight (valuation.ts, the C300 case). The headline was new cars all along.

Recorded because the lead is genuinely attractive from metadata alone and will be found again. Counting which keys exist is not inspecting what they hold — the same mistake shape as DATA-QUALITY 35, where a counter's presence was mistaken for its responsiveness.

38A guard that skips must not exit clean

crawl.ts refuses to start beside another crawl — correct, and the reason per-host politeness holds at all. It then returned, exit 0. Fourteen lines above that return sits a comment describing a pass that "exited having done absolutely nothing — while the workflow went green, which is the worst way for this to fail."

On 2026-08-17 it did it again. The deep pass started 15:43:30, ran the whole chain, filed a green 21-check integrity run at 15:45:27 and stamped its heartbeat ok=true. 118 seconds, ~600 pages against the 18,000–22,000 a real deep pass fetches, and no delistings for another twelve hours.

run-chain.sh already knows a deep pass must not skip — it waits up to two hours, because losing a deep slot costs twelve hours and every delisting in them. But that wait is driven by pgrep, which sees only that box. A crawl running anywhere else — a laptop, a hand-run sweep — is invisible to pgrep and visible only to the database guard. So the deep pass walked past the wait it was entitled to, tripped the guard instead, and returned success.

Measured before fixing, over twelve days: two of the box's deep windows fetched ~2,000 pages and recorded zero delistings, against 18,000–22,000 pages and 250–700 delistings for the ones that ran. About one deep pass in six, each silently, each costing twelve hours of the only evidence that a car has sold.

So the wait moved to where it can see: into crawl.ts, the one choke point all three scheduler paths share, bounded at ninety minutes and re-reaping abandoned rows each round so a crawler that dies during the wait cannot hold the slot. The shell keeps its pgrep wait; it simply is no longer the only one.

The asymmetry then reaches the exit code for the case where waiting was not enough: a fast pass really does catch the next slot two hours later and still exits clean; a deep pass that gives up exits non-zero, failed=1, the heartbeat records ok=false, and health.ts turns that into an email.

Third occurrence of one shape — a component reporting success for work it did not do (the 41ms deep unit, the unread RPC error that printed sources=0/696, this). The lesson that keeps not sticking: the exit code is a claim about the work, not about whether the function reached its end.

Health gained the matching check at the same time. Every other signal reads the corpus, and a corpus looks healthy for days after crawling stops — 200,000 live listings sit there being live, boards stay fresh because pg_cron computes them in Supabase. Only the absence of a heartbeat says the machine has gone, and nothing was reading it. Each kind declares its own period_hours, so one rule covers a two-hourly fast pass and a twelve-hourly deep one: two and a half periods of silence, because one missed run is a reboot and two in a row is not.

39An index can stop enumerating without ever going down

Westside Auto Wholesale — ~1,900 cars, one of the largest single sources in the corpus — reported complete=true on every deep pass while finding exactly 18 cars, for five days straight. Nothing errored. The crawl fetched 419 pages a pass, the site answered 200 to everything, and 1,905 listings sat "live" with ageing prices because the delist check re-fetched their detail pages, got 200s, and correctly refused to call them sold.

The site had redesigned around Aug 17/18: /cars and /cars?page=N now both server-render only the first 18 cars — pagination went client-side, so every page the crawler followed carried the same 18. From our side it looked like a yard that had simply stopped trading, which for a 1,900-car wholesaler is not a plausible state of the world.

The repair was one config row, not code: their robots.txt publishes sitemaps/individual-cars.xml with 1,803 car URLs (verified), the carma pattern exactly. indexUrl moved there; /cars stays as a secondary index so fast passes still touch the top of the yard.

What earns the entry: found-count collapse is a signal no check was reading. A source that went 1,795 → 18 overnight and held there is not subtle — but complete=true and HTTP 200 everywhere meant every existing check saw health. The number that knew was listings_found against its own history. Also the timing: the collapse landed the same night a code deploy did, and the code was innocent — ?page=2 returning page 1's cars settled it. Correlation with your own deploy is where you look first, and evidence from the site itself is what decides.

40The pipeline was on a website visitor's leash

Every night since Aug 18, the deep chain's tail failed with "canceling statement due to statement timeout": compute-insights died mid-corpus-read, check-integrity died on check_delistings_from_partial_runs, and no scheduled integrity row was filed for five days — health said "integrity suite has not run for 136h" while the crawl itself was healthy.

Neither query is slow. The delisting check runs in 955ms warm (EXPLAIN ANALYZE, 2026-08-23); the insight corpus pages at ~1.4s each. They die because PostgREST requests inherit the authenticator's statement_timeout=8s unless the impersonated role sets its own — anon 3s, authenticated 8s, service_role nothing. Our own batch pipeline ran on the same 8-second leash as an anonymous page view, and the corpus doubling in a week pushed its heaviest statements past it exactly when the box is busiest.

Migration 0196 gives service_role 120s. The public site's limits are untouched. Same class as DATA-QUALITY 16 (the contributing_sources RPC timing out under load) — that entry made the failure loud, this one removes the false ceiling. pageByKey also now retries a page that hits a statement timeout, halving the page size each attempt: the corpus read paged for five minutes and lost everything to one bad page, and keyset pages cost roughly linearly in rows, so half the rows is half the statement time. Only timeouts are retried; every other error still throws on sight.

41When a platform vendor ships a theme, twenty yards break at once

Nineteen dealer-studio sites collapsed to exactly listings_found=21 — one index page — on exactly 2026-08-16. Not gradually: 559→21, 513→21, 368→21, overnight, across unrelated dealer groups in three states. No extractor commit landed in the window; Dealer Studio rolled a theme that moves index pagination client-side to a cohort of its customers, and every one of their yards went dark the same way Westside did (DATA-QUALITY 39) for the same reason: the crawler paginates, every page renders the same first 21 cars, complete=true, and 3,716 listings age silently while their detail pages answer 200.

The repair is also the same shape, at platform scale: the new theme publishes sitemap-custom.xml holding the full /cars/ inventory — verified on all 19 sites individually (3,906 car URLs total) before touching any config. indexUrl moved to the sitemap on all 19; the old index page stays as a secondary entry so fast passes keep touching the newest stock.

Two lessons over 39's. First, a platform is a correlated failure domain: one vendor decision broke 19 sources simultaneously, and the corpus holds 465 dealer-studio sites — when the vendor rolls this theme to the rest, the same fix applies, and the signature (found pinned at exactly one page, same day, many sources, one platform) is now known. Second, the audit that found this started from a warning, not a failure: "15,882 live listings unseen for over a week" decomposed into marketplace window-churn (10,700, structural, documented) and this (3,716, a real break). A warning that is never decomposed is a warning that will be ignored.

42The client noticed before we did

Kiel checks Westside's yard page daily. For five days it did not change, because the crawl behind it was blind (39) — and the first the system heard of it was his feedback. Every alarm we had watched the corpus, and the corpus looked fine; nothing watched a single source against its own history, and nothing gave a dealer with a portal account any more freshness than a dealer without one.

Two changes, one lesson each:

check_found_collapse (migration 0198, wired into health): a source whose best found-count over the last two days is under a quarter of its best over the prior week, floor 40, alarms — and a source with no recent runs at all counts as collapsed to zero. Run against history it would have caught Westside and the dealer-studio cohort on day one; run today it immediately surfaced three breaks the manual audit had missed (buckbyram and patrickautotraders and cheapcarstas — imotor's new Algolia-backed theme, pending a proper extractor; tonylahood — easycars moved its index, fixed with one config line). A detector pays for itself the first time it runs.

crawl_config.priority — a dealer with someone watching gets fast passes that do the deep pass's job: full index traversal (for a sitemap source, one XML fetch), a 40-page detail budget the existing rotation spends on never-seen cars first, and complete left honest so the delist check may run. Verified live on a fast pass: found=46 against the one-page 21, new=1 caught, delist check ran and correctly refused a still-200 candidate. Delisting safety was not moved: it rests on presentUrls from the traversal, the same invariant --max-detail documents, not on how many detail pages were fetched. Exits now reach the portal within a couple of hours instead of within twelve.

The lesson over 39/41: the corpus being healthy and every source being healthy are different claims, and a client-facing page is a per-source claim. Monitoring must match the granularity of the promise.

43The unlinked sitemap, and not building what you don't need

The three imotor sites that collapsed on the vendor's Gatsby+Algolia theme (42) looked like they needed an ingest of their own: every HTML route serves the same 12 cars, the inventory sits behind the site's own Algolia index (dealer_stock_<id>, nbHits visible in the page's serverState), and the first instinct — recorded in a spawned task — was to read that backend the way Dutton One reads AppSync.

The instinct was wrong, and checking the cheap option first is why it cost an afternoon less. imotor publishes sitemap-stock.xml on every one of these sites — buckbyram 242 car URLs, patrick 81, cheapcarstas 43, each exactly matching its own nbHits — holding the full inventory as ordinary detail URLs the existing extractor already parses (verified: a 2024 GLA250, $52,990, 22,974km, VIN read clean). It is simply not linked from `sitemap-index.xml`, which lists only the 39-URL marketing sitemap. The fix was the dealer-studio config row (41), not a line of new code, and the Algolia reverse-engineering — chasing an obfuscated app key through the minified client — was abandoned the moment the sitemap turned up.

Two things worth keeping. First: the credential hunt that isn't needed is the cleanest credential hunt. SPEC's honesty rules made the Algolia path legal, but a static sitemap the site publishes for search engines is simpler, more stable, and asks the origin for less. Reach for the API only when the HTML genuinely holds nothing. Second: 114 of the 115 imotor sites already pointed at sitemap-index.xml were healthy, so their index does link their stock sitemap — buckbyram's cohort were the ones still pointed at the HTML /stock page, which is exactly the surface the new theme hollowed out. The break was in which entry point we'd chosen, not in the platform.

44The pulse bar took the whole site down

The site went slow-to-unresponsive: /value and /sign-in took ~19s, /coverage timed out, while /market stayed 0.24s. The split was the tell — /market reads precomputed boards; the slow pages didn't.

Every page renders the credibility bar from market_pulse() in the root layout, so a slow pulse is a slow everything (that is why /sign-in, which touches nothing else heavy, was 19s). market_pulse was a live scan — sold_total counts every delisting ever recorded, with a correlated NOT EXISTS per row and a listing_event join. At its documented 104ms nobody noticed. Then today's own fix drained the backlog: recovering the collapsed sources (39/41/43) wrote hundreds of delistings at once and left the tables full of dead tuples, and the scan tipped past what fits in the Small instance's cache — 30-100s, reading from disk.

Then it cascaded. The in-app cache (cache.ts) protects a warm process with single-flight, but every cold Vercel instance pays the full cost on its first caller. At 100s a caller, pages hang, Vercel spins up more instances, each starts its own market_pulse, and ten concurrent disk-bound scans of a corpus that no longer fits in cache is a site that does not load. pg_stat_activity showed exactly that: many concurrent authenticator market_pulse calls, all on DataFileRead/BufferIo.

Fix (migration 0200): the pulse only moves when a crawl writes, so it reads a single-row board refreshed every 5 minutes by pg_cron — the shape /market, coverage (0154) and the depth boards already use. market_pulse() went from 30-100s to 0.75ms. Deliberately no live-scan fallback when the board is briefly empty: a scanning fallback is the stampede coming back. Applied as a DB function, so it fixed every cold instance at once with no deploy.

Two lessons. First: a fix that recovers a lot of data is itself a write spike, and a request-path query that was fine at one corpus size is not fine after a backlog lands. Second: anything the root layout awaits is on the critical path for the entire site — market_pulse should never have been a live full-corpus scan there, cache or no cache, because the cache does not cover cold instances. discount_by_time_on_market and turn_segments are live scans on the same footing; boarded here or not, they are the next candidates if they ever show up slow.

45Boarding the pulse's two siblings — and the view that scans everything

The two live scans DATA-QUALITY 44 named as next candidates — discount_by_time_on_market and turn_segments, both on /market — got the board treatment (migration 0201): a single-row jsonb board the read function expands back into its table shape, refreshed every 6h by pg_cron, read in 0.6ms on the request path.

The trap was in the refresh. Both original functions read from listing_current where status = 'delisted', and lifting that verbatim into the refresh took 311 seconds — because listing_current runs a price-change-count subquery and a double lateral per row, and the planner computes all of that for the whole 257k-listing corpus before the status='delisted' filter removes 87% of it. The board would have swapped a slow request-path scan for a slow cron scan.

Reading base tables instead — listing filtered by listing_status_idx to the 34k delisted rows first, then observation_pick_current_idx for each row's final fields — dropped it to ~107s. Still not fast (34k random reads into a 3.9M-row table on the Small instance), but off the request path and safely inside the cron budget. The lesson for reuse: a heavy view plus a WHERE filter is not a filtered heavy view — the per-row work runs before the filter unless the planner can push it down, and over listing_current it cannot. Aggregations over a subset of the corpus should read the base tables, not the convenience view.

46The fallback was half the data

The sold-car facts table (0202) unified four request-path scans — recent_sales 115s, turn_filtered 53.7s, turn_by_model 53s, days_to_turn 2.2s — into one half-hourly rebuild and millisecond reads. The build skipped listing_current's fields_empty fallback, citing DQ45's "fraction of a percent". Wrong population: across ALL listings the fallback is rare, but a DELISTED car's final observation is very often an empty-fields sighting — a delist probe, an index pass — whose real payload sits one fields_sha lookup back. The view dates 8,133 sold cars; the fallback-less build dated 4,386. turn_by_model shrank from 305 models to 218 and the shipped 0201 boards undercounted the same way.

Caught because the row count was compared before anyone consumed the numbers, fixed in 0203 by carrying the view's recovery lateral verbatim into the build (8,109 of 8,133 after; the residual 24 is the deliberate stricter "any live listing anywhere blocks sold" check). The 0201 boards now read the facts table — correct AND retiring their duplicate 107-second scans.

The lesson, sharper than DQ45's: a shortcut validated on the general corpus was applied to the one population where it fails hardest. Delisted cars are not a random sample of observations — the very event that makes them interesting (leaving) is what empties their final payload. Verify a simplification against the exact rows it will serve.

47Green means every check can actually run

Confirming "everything is green" after the facts-table work surfaced three more members of the outgrown-scan family, each flapping rather than failed — passing on a quiet afternoon, dying under crawl load, which is the alarm shape nobody trusts:

check_fields_empty_flag scanned all 4.19M observations three times per run and killed the deep chain's integrity step at midnight. Its gate — "mislabelled must be zero" — now counts a partial index built on exactly the violating predicate (fields_empty and fields <> '{}'): maintained free by every insert, covers all history, microseconds to read. Stronger than the scan it replaces, not weaker. The first windowed rewrite (3 days) still measured 116s — a window over millions of jsonb rows is a smaller full scan, not a different shape; the companion info counts settled at a 6-hour sample (2.6s cold). Migration 0204.

run_slow_integrity_checks timed out at 12:45, 18:45 and 00:45 — every slot the crawler or a hand-run sweep held the disk — leaving three checks reading 20-hour-old answers against a 12-hour limit. Moved to the quiet windows (01:45, 13:45), where its answers are at most ~9.75h old when the deep chains read them. Migration 0205, alongside depth-boards' identical move (02:25/12:25) after refresh_live_snapshot lost the same IO race.

Two mirror pairs returned — mtfordmtbarker shadowing mountbarkerford, hurstvilletoyota shadowing fergusonstoyota, 39 live URLs under two ids. 0083's re-runnable merge procedure resolved both in place; the host rule picked the NEW domains as canonical, which is the tell that these were rebrands discovery correctly found before anyone retired the old ids.

The pattern across the night: the corpus doubling did not break one thing, it moved every full-scan's cost past a different limit at a different hour. Fixes that reshape (facts table, partial-index gate, boards) beat fixes that re-budget, and a scheduled heavy job belongs in the hours the crawler is provably quiet.

48Green, verified from both sides of the event

The convergence held: the partial index built in 228s, the VIN check alone ran in 158s (it died past 900s when bundled), the windowed slow checks in 91s. Then the full integrity re-run surfaced the day's last two truths.

The 2026-08-23 daily rollup is absent, permanently and honestly: its 15:55 cron collided with the 15:44 deep pass and died, and write_daily_rollup's own guard refuses to describe a past day with today's snapshot — the same discipline that left 08-12 empty. The job moved to 12:50, inside the quiet window, and stopped double-refreshing the snapshot. Two gap days now stand in the series as the record of two outages.

And the delist check learned what a merge and a killed crawl already knew: an event is justified by a complete traversal on EITHER side of it. A merged pair shares its history (the run that justified an event may belong to the id its listing now files under), and a delisting whose car stays gone through eighteen subsequent complete traversals is proven by absence — a fabricated one would have been relisted on sight by the first of them. Eight orphaned events from the campaign week's killed crawls closed out on that rule; a fresh fabrication still alarms immediately, because it has no complete traversal on either side. First fully green suite: 21 checks, two known work-queue warnings (156 VIN pairs, marketplace window staleness).

49Lapsed: the fourth status, for cars we can no longer watch

Integrity 17 went red on 08-25 and stayed red: 10,986 live listings unconfirmed for a fortnight. Decomposed: 9,742 behind autotrader's and carsguide's moved search windows (never re-seeable, growing forever), 397 on three sources deliberately withdrawn on 08-15 (right call, but it stranded their listings as "live" for good), and 847 at keep-the-page-up dealers whose cars the delist probes re-verify every few days without the freshness clock ever hearing about it.

Three truths, three fixes. Lapsed (0208) is what "live" cannot honestly claim and "delisted" must not: watching ended. Withdrawn sources' listings and marketplace listings unseen 14 days lapse nightly; they leave the live corpus, never enter the sold tape, and relist on any sighting — nothing ever said they went. Dealer listings never lapse on index absence alone, because our own broken enumeration would mass-lapse real cars (vonbibra serves 24 of its 223; that gap is ours). Probe sightings now stamp last_seen — the delist check's verdict "still answered 200, not sold" keeps a car live on evidence the freshness clock never saw. And the Penrith unknown-unknown class got its systemic fix: the dealer-studio extractor falls back to the dealer's own URL slug when the DMS payload says Unknown — the fourth recurrence in three days was the sign hand-edits were a treadmill.

Also: check_found_collapse's prior window widened to 30 days (0209) — an enabled source silent longer than the old 9-day window vanished from the check entirely, and silence past the window must read as more collapsed, not less.

50A rebuilt site keeps its cars behind an API, and a rebuilt address is still the same car

Ringwood Nissan rebuilt itself on 09-01: a Next.js app whose sitemap lists marketing pages and offers but no cars, whose /stock page is a client- rendered shell, and whose inventory arrives from /api/stock?page=N — the site's own public JSON, permitted by robots.txt. The old sitemap 404'd and 164 live cars went dark; the collapse alarm caught it in the next health run, as it should. The nextjs-stock-api extractor makes that endpoint the source's index: the ordinary crawl reads it through politeFetch, every advertised car arrives inline, and run rows, captures, price changes and complete-traversal delisting follow with no new machinery. Two honesty rules fall out of the shape: detail pages return the same "No Cars available" shell for a real car and a bogus slug, so no page can ever confirm a car — safe, because the crawl only confirms delist candidates, cars a complete API traversal no longer lists; and the API keeps returning cars it has sold (status 1, sold_at), which are not advertised and are not emitted.

Four hand-run passes: 152 of 168 records advertised, 152 live rows, and the old rows retired on the 404 the rebuilt site serves for its old addresses — which the review then showed was the wrong reading. A 404 on every address at once is evidence about the site, not the car. 150 of those retirements were re-recorded as lapses (0222): watching ended, nothing said the car went, and thirteen of them were still on the yard under new addresses the old rows could not be linked to — on the sold tape for half an hour as sales that never happened. The delist check now lapses rather than delists when 404s arrive above the soft-404 blast radius, and a sold car on this shape leaves the corpus by the seller's own declaration in the feed (status 1, sold_at), the only signal its 200-shell pages will ever give.

Two costs stated rather than hidden. The lapse swept up two cars the feed itself lists as sold (an i30 and a Patrol, sold_at 29 August); their rows sit at pre-rebuild addresses the feed-sold sweep can never name, so those two real sales are not on the tape. Two hand-edits could put them there, and a hand-set row is the thing entry 4c exists to catch; they stay lapsed. And the first version of the blast guard was denominated in live rows while the probe budget is a quarter of the candidates — unreachable for any yard over ~240 cars, which is 65% of the corpus. It is denominated in probes answered now: most of the yard gone from the index at once, and nine in ten answers a 404, is a changed site; a quiet dealer's clear-out is neither.

Two things the rebuild taught about identity. First, a reused row must follow its advertisement. observeListing resolves a listing by url, else by stock number with make and model agreeing, and by design never rewrites what the row already says — so sixteen cars whose stock numbers survived the rebuild were correctly reused and left holding their OLD url, which 404s and is absent from every present set: a delisting confirmed by the dead page, a relisting on the next index read, every pass. The move now lives in the crawl (not in observeListing), gated on the one fact only the traversal knows — the stored url absent from a COMPLETE present set while the observed one is in it — which is what separates a moved advertisement from one car genuinely advertised at two live addresses, of which the corpus holds 1,805 rows.

Second, what cannot be re-identified is not merged. 22 of the retired rows carried neither VIN nor stock number, and each has a live VIN-bearing twin of the same make, model, year and odometer — but so do forty others: they are new-model stock at ~10 km, and make/model/year/km cannot tell one 2026 X-TRAIL from the next. The pairing is 22 × 105, not 22 × 22. Merging on that would be a guess wearing a vehicle id, so the rows stay separate: the old advertisements did end, the retirements are true, and up to 22 vehicle rows may be duplicates of live ones — stated here rather than resolved by fabrication. The general lesson is the one 0173 already drew: the VIN is the car; a row without one cannot cross a rebuild.

51The fix for the row cap outgrew the row cap

/coverage said 1000 crawled against a true 1,044, and 44 dealer sources rendered as "not yet crawled" — 55,410 live adverts, 24.3% of the corpus, including Autotrader, CarsGuide and the pilot dealer Westside Auto Wholesale, every one of them crawled within the hour.

The bitter part is the provenance. latest_source_runs() (migration 0081) exists because of this defect. Before it, the page fetched every crawl_run and reduced in TypeScript, which broke once that table passed PostgREST's 1,000-row cap; 0081's own header records the diagnosis and even records that limit(8000) did not help, "the ceiling belongs to the server".

Moving the reduction into the database changed how the rows were computed, not how many came back. The function returns one row per source. Sources passed a thousand on 2026-08-18, and from that morning the call returned exactly 1,000 rows with no error, for a fortnight.

Why nothing caught it

Every symptom looked like a fact about the world rather than about the read. A source with no run row renders "not yet crawled", and the page's own copy explained that as "it has not been visited" — a sentence that is true of a newly seeded source and was false for 44 of the 46 rows carrying it. The sources that fell off were the tail in source_id order, which is not a suspicious set. And the tell was a round number in an eyebrow that renders without a thousands separator, so 1000 did not look round.

The rule this earns, on its fourth recurrence

Entries 9, 10, 21 and 26 are the same ceiling. What is new here is that the remedy inherited the defect, so the rule has to be about the read rather than about the query:

A read is either provably smaller than its ceiling, or it is paged. "I moved the work into the database" is not the same claim as "the result fits", and a function that returns one row per anything is a function whose result grows with that thing.

Fixed, and what the fix cost

/coverage pages the RPC by key through pageByKey. PostgREST applies filters and ordering to a set-returning function's result exactly as it does to a table, so no SQL changed.

It is not free, and the cost was measured rather than assumed. A SQL function carrying set search_path cannot be inlined, so the cursor is applied after the function has materialised all 1,044 rows: each page is a full execution, roughly 280,000 buffer hits and 0.5 to 1.6 seconds, and the page now pays two of them. The right end state is a board — coverage_board already exists and is refreshed every fifteen minutes — or p_after/p_limit arguments so the distinct on stops early. Correctness first, then the board; recorded here so the second half is not forgotten.

Two siblings found by the same review, one of them live

/api/statements/chart-proof carried .limit(3000) on a listing_event read — inert, because the server caps at 1,000 — with the error discarded. Westside's August holds 1,093 price events, so the proof sheet's cut-dollar series was built from the first 1,000 and drew $990,601 against a true $1,084,201, understating the month by 8.6%. Both of its reads are now paged and neither swallows its error. The issued statement PDF was never affected: it reads a different path, and its own $1,084,201 is what showed the proof sheet was wrong.

pageByKey also gained a guard. Its stop condition is "a page came back shorter than I asked for", which cannot tell the end of the data from a truncation — safe at 1,000 and only at 1,000, because that is PostgREST's db-max-rows. A pageSize above it would have returned a truncation as a complete answer through the helper written to prevent exactly that. It now throws instead. The temptation was real rather than theoretical: the paging cost above makes "just fetch 2,000 at a time" the obvious next edit.

Still unpaged and worth watching, none losing rows today: source_max_found() returns exactly 1,000 right now and decides every source's detail budget; pass_ordering_inputs() is already over the cap and is read unpaged. Both are one source away from the same silence.

52Two sources the box could not reach, and one symptom hiding two facts

Health alerted on 2026-09-04 that dealer:berwickldv had found 0 listings in two days against 56 the week before. A query for the whole class — enabled sources whose last five runs all fetched nothing with a fetch error — found a second, dealer:grandprixautogroup, which the alarm's two-day window had not yet reached. Both showed the box the same thing, TypeError: fetch failed on the index, five for five. The facts behind them were different, and only evidence from outside the box could tell them apart.

Berwick LDV is gone. berwickldv.com.au and www.berwickldv.com.au answer NXDOMAIN from both 1.1.1.1 and 8.8.8.8; the site last answered 200 at 2026-09-01 23:23 UTC. None of its 36 VINs appear at any of the seven Berwick sibling sources, so the stock did not move somewhere we watch. The source is withdrawn with the evidence in robots_notes — disabled, not deleted, like every withdrawal on the OPERATIONS table.

Its 36 live rows (32 new, 4 demo) are lapsed, not delisted. The crawl's own invariant is right and was left alone: a failed fetch proves nothing about a car, so the delist check changes nothing on one — which is exactly why those rows would otherwise have sat "live" with a 1 September last-seen forever, while the site showed them as stock. A domain that has ceased to exist is evidence about the site, not the cars, and that is what lapsed (0208) exists for: watching ended, nothing said any car went, none enters the sold tape, and every one relists on sight if the domain comes back. The reason string joins the 15,931 lapses already recorded under "watching ended: source withdrawn". recordLapse was exported from store.ts so the script that did this (scripts/repair-unreachable-sources.ts) writes the same two rows the crawl writes, not a copy of them.

Grand Prix Auto Group is reachable from here and, mostly, not from the box. Its index sat on the apex, https://grandprixautogroup.com.au/used-cars, which the box has failed to fetch on nearly every run since 2026-08-30 — while a capture from www.grandprixautogroup.com.au/search/used-cars succeeded on 2026-09-03 and all 28 of its live rows carry www addresses. The first repair read that as apex-versus-www and moved the index to the www URL the apex itself redirects to. That was not the fix. The box's first pass on the new URL, at 11:21 UTC, failed the same way, and the run history says why the premise was wrong: the box has reached this host about once in thirteen attempts every day since 30 August — one success in 13, 14, 14 and 13 runs on consecutive days, on either address — so the capture that looked like proof was the one attempt that got through. From a laptop, through politeFetch, both addresses answer 200 in seconds, every time.

A DNS census of every enabled dealer host settled what it is not. 179 of 999 resolve to 216.150.x.x and 331 to 76.76.x.x — 510 sites on the same Vercel infrastructure, including this one's neighbours on the identical apex IP — and the box crawls them normally. So it is not "the box cannot reach Vercel". It is this host, from that box, and the reason was unknowable from the record because every connection-stage failure reaches the crawler as the one string TypeError: fetch failed with the real error on cause — DNS, the SSRF guard's refusal, a connect timeout, TLS — and the run rows stored the string. Seven days of failures could be counted and not one could be named. The crawler records the cause now, code first, so the next failure says what it is; the index stays on the www URL because it is what the apex redirects to and it changes nothing either way.

The first recorded cause arrived at 17:19 UTC the same day, and it names the fault exactly: getaddrinfo ENOTFOUND www.grandprixautogroup.com.au. The box's resolver cannot resolve this host --- a numbered vercel-dns-016 CNAME chain --- while 1.1.1.1, 8.8.8.8 and a laptop resolve it and the site answers. The SSRF guard resolves through dns.lookup, the system resolver, and rightly refuses a host it cannot resolve, so an upstream resolver that fails intermittently on this chain produces precisely one success in thirteen. The fix is on the box, and the next question is the class: any other ENOTFOUND in a day of crawl_run.errors is coverage the resolver has been silently costing. The 28 live rows are 40 hours stale and the collapse alarm will name them tomorrow, correctly; they are not lapsed, because the site is up and the cars are advertised — the fault is on our side of the wire and the record now has what it needs to find it.

Three lessons beyond 39's.

The alarm's window is the alarm's blind spot. check_found_collapse needs two quiet days; a class query on crawl_run needs five failed runs, which at this source's cadence is ten hours. Run the class query whenever the alarm names one source — the second one was already there.

A hand-run pass cannot take the mutex, and should not try. The deep-pass guard (anotherCrawlIsRunning) waits a minute at a time for the box's continuous crawl to end, which it never does; a laptop pass for one source sat in that loop for seven minutes doing nothing, correctly. The proof of a config repair is the box's own next run, and that is the better proof. The --fast form steps around held sources, but two crawlers on one host is the DATA-QUALITY 18 risk, so it was not used either.

One job's deadlock was another job's lost day. daily-rollup at 12:50 found live_snapshot 6h45m old on 2026-09-03 because depth-boards had deadlocked at 12:25 (OPERATIONS, "A board rebuild deadlocks…") and, being one transaction, rolled its own snapshot refresh back with everything else. The rollup's refusal is correct — a day's stock columns can only describe that day's snapshot — so 2026-09-03 is a gap in all five daily series and stays one; writing it from the 4 September snapshot would blend two days into one row. The coupling is gone instead: daily-rollup now refreshes the snapshot itself when it finds one over 90 minutes old (migration the_daily_rollup_refreshes_the_snapshot_it_cannot_find), which costs nothing on a normal day and a few minutes on the day it matters. The original split (0111) was made when the pair blew a 15-minute ceiling on the 8-second leash 0196 later removed; measured now, the rollup alone is 11 seconds.

Resolved 2026-09-05 (07:21 AWST). politeFetch now asks 1.1.1.1 and 8.8.8.8 when, and only when, the system resolver answers ENOTFOUND, runs the same SSRF guard on the answer, and logs the fallback once per host. The box's first pass after the deploy (23:21 UTC on 4 September) fetched the Grand Prix index and found 21 listings with no error, where the pass two hours earlier had failed with the same ENOTFOUND as every pass since 30 August. Any other resolver failure is still returned as it came.

53The status walk and the board rebuild took the same boards in opposite orders

depth-boards deadlocked four times in a fortnight and lost a full day of boards on 2026-09-03, then the 3 September daily rollup with it (52). Every instance had the same shape: the job held one board and wanted another while some reader held the second and wanted the first — aged_board against live_snapshot twice, model_chart_board against model_event_board twice. Postgres picked the writer as the victim each time, which is the expensive choice: a twenty-minute rebuild dies so a status query can finish.

Two guesses were wrong before the facts were. The first chip blamed "a page reading two boards"; no view or RPC reads both members of either pair. The second suspected a cron job; no other job reads those boards at all. Both searches looked for literal table names, and the reader builds its table names at runtime. board_status() — the one RPC every /market and model page calls to date its figures — walks board_spec ORDER BY board in a single transaction, execute format(... from public.%I ...) per board, holding AccessShare on each until the last is counted. Alphabetically aged_board precedes live_snapshot and model_chart_board precedes model_event_board; the job truncated each pair the other way round. That is the entire mechanism, and the only reason it was two pairs rather than seventeen is that the job's order happened to agree with the alphabet everywhere else.

The fix is one statement: the job takes AccessExclusive on all seventeen boards it will truncate, first, in the alphabet's order. A writer that holds nothing cannot be part of a cycle when it waits; once it holds everything, any reader waits on its first board and never holds one the job wants. It does not hold anything longer in a way anyone can see, because live_snapshot — the board every reader reaches — was already taken in the job's opening seconds and held to commit. The reader is untouched: the authenticator role already caps a lock wait at eight seconds and board-status.ts renders undated on any error.

The guard matters more than the fix. A lock list is exactly the kind of thing that drifts — the next migration to add a truncate to one of those six functions reopens the cycle without touching the cron command, and nothing would say so until the job died again. check_depth_boards_lock_order() recomputes what the job's functions actually truncate and demands the command lock precisely that set, alphabetically. Proved to fire on a synthetic list that locked one board of seventeen: sixteen missing.

A coda from the fix's first scheduled run. It succeeded --- 1,166 seconds, seventeen locks taken at the door, nothing stuck --- and the deep chain, whose crawl had run until 12:38, placed check-integrity inside the window. Its freshness check reads board_status(), waited on aged_board until the authenticator's eight-second lock_timeout cancelled it, and threw before filing a row: the chain reported "a step failed" with nothing to say why. A freshness read has no business waiting on a rebuild at all --- a locked board is not a stale one --- so board_status() now gives up after two seconds and both suites treat that as "rebuilding": a warning, never a false ok, and the verdict is still filed. The collision predates the lock; it needed a deep crawl long enough to land its tail in the window, and today's was.

Two smaller things. deadlock_timeout is 1s and the writer waits on a lock long after the reader started, so detection always ran in the writer's process; that is why the job was always the victim. And a lock table of many relations acquires them in the order written, which is why the order is spelled out rather than left to the truncates — the whole defect was an order nobody had written down.

54Seven functions were open to the browser roles for a day

What. Between 2026-09-04 and 2026-09-05 six new SQL functions (the make and segment tables, the state profile and its year overload, the aged-share benchmark, the report series wrapper) and one recreated one (the API quota check, dropped and recreated to return the limit) were created without the revoke execute ... from public, anon, authenticated every other function carries. Postgres grants execute to public by default, and a recreated signature starts from that default again. The site reads them through the service role; the browser never called them. The functions are read-only aggregates over public data, except the quota check, which writes an api_request row per call --- an anonymous caller could have logged requests against a key's sha-256 had they held one.

How it was found. The integrity suite's "every public table is closed to anon and authenticated" check, run by hand after the day's work, listed all seven. The suite runs on the box's chain and on GitHub Actions every six hours; the window was under a day.

Fix. seven_functions_added_this_week_are_closed_to_the_browser_roles revokes execute on all seven. docs/OPERATIONS.md now states the rule beside the migration idiom, and the check stays as the net.

55Five reasons a dealer's car stayed live after it had gone

What. On 2026-09-05 the integrity suite warned that 9,830 live listings had not been seen for a week and 629 dealer asks were unconfirmed for a fortnight. Decomposed by source, the first number was mostly by design and the second was five separate faults, none of them a dealer selling slowly.

• 8,344 of the 9,830 were autotrader and carsguide rows aged 7-14 days. A marketplace row is lapsed at 14 days unseen (entry 0208) and nothing about its 7-14 day tail is wrong; the warning counted it anyway, sat permanently on, and hid the dealer count it exists to show.
• Maddington Isuzu UTE probed 58 dead cars a run, every one redirecting to /stock with 24 other cars on it, and recorded none as gone. The stock-page rule read found.inline?.length ?? found.listingUrls.length; the JSON-LD extractor returns an empty inline array on every page, so ?? answered "zero other cars" on a stock page carrying twenty-four links. The rule had never fired on any of the 74 jsonld sources. 1,827 such pages were probed across them in the previous 24 hours.
• Tynan RAM, Rockdale City Chery, Omoda Jaecoo Gosford and four more iMotor sites read 91% on the parser-health gate and were refused the soft-404 path every run. Their sitemaps list /latest-news/<slug>/<id>/ beside the cars, the generic sitemap rule accepts any nested path ending in four digits, and the extractor's own precise list was unioned with it because the generic list was larger. 264 news pages a day were fetched as cars and counted as parser failures.
• Thirty-three iMotor sites of the server-rendered flavour read 0/0 on the same gate. That flavour renders page 0 of the stock inline and fetches no detail page, so the gate had nothing to measure and refused every run; Queanbeyan Toyota held 130 cars live against 12 on its index.
• cars24 keeps a sold car's page, title, photos and price, and disables the buy button with the label SOLD. No title rule or phrase matched it, the Vehicle node is gone so the page reads as "no car", and cars24's detail parse rate of 75% (the sold pages are the failures) refused the soft path.
• BMW Sydney moved to sydneybmw.com.au. The old sitemap index names a child that answers HTTP 500, so no traversal had completed since 2026-08-12 and the delist check never ran. Evergreen Auto answers every /stock page with HTTP 200 and an empty body to the honest user agent, while its home page and sitemap answer normally; 204 URLs found and 0 extracted, every run.

How it was found. Reading the two warnings per source, then fetching one stale listing from each of the worst sources through politeFetch and watching what the extractor, the sold reader and the shape fingerprint made of it.

Fix. || in place of ?? for the other-cars count. Extractor.isListingUrl, which iMotor implements from its detail-path pattern; a sitemap loc that fails it is never fetched, and the generic rule and its union are skipped for a platform that states its shape. The health gate accepts twenty probes that read a car off the car's own page as the measurement when the detail loop had nothing to measure, and still refuses a loop that measured and found the parser flaky. A fourth sold-marker kind, control: a disabled button whose whole text is SOLD. Integrity check 6 counts dealer rows only and is renamed to say so; marketplace rows overdue for their lapse remain check 17's at 15 days. Sydney BMW's index is its stock sitemap on the new host. Evergreen Auto is withdrawn with the reason on the source, and the nightly lapse marks its 255 rows as watching ended. First run of the fixed rule on Maddington Isuzu UTE, 2026-09-05: 60 candidates probed, 56 no longer describing a car, 45 of them gone three days or more and served the stock page, 45 recorded as no longer listed (confirmed_via: soft 404, never as sold); detail parse 346/346.

Addendum, 2026-09-06. The integrity check "every soft-404 delisting carries the basis it rested on" failed overnight on 98 rows: the events written under the probe-measured health rule record detail_parse_rate 0 (the detail loop fetched nothing) beside probes_read_a_car of 23 to 54, and the check read the 0 as a dead parser. The rows are legitimate; the check now accepts either measurement (the_soft_404_check_accepts_the_probes_as_the_parsers_health) and each event names which one it rested on (parser_health_basis). The Health workflow's one failure the same morning was this alert relayed.

56A one-yard account read seven cars standing, and four days of red

What. Scarboro Toyota, linked to an account through the group model on 2026-09-07, showed 7 cars standing against 396 live. The branch row carried dealer_name = 'Scarboro Toyota', and the group functions filter listings by that name where it is set; the yard's listings carry three names ("Scarboro Toyota - Used Cars", "Scarboro Toyota - Used Cars (304)", and plain "Scarboro Toyota" on seven). The filter exists for the one-site, many-branches case (Lexus of Perth) and must be null for a branch that is a whole source. Set null on 2026-09-10; the group position now reads 395 cars over 390 dated.

The integrity suite failed every run from 2026-09-07 04:29 to 2026-09-10 04:33, three causes in turn:

• 2026-09-07 to 09-09: "every yard with a portal account is crawled as a priority" --- the new membership had no priority flag on its source. Set 2026-09-10; the Health workflow's failures those mornings were this alert relayed.
• 2026-09-10: the capture-order check timed out at 120s. It walked every observation of the last 36 hours joined to raw_capture, about 800,000 rows once the corpus passed ten million; it only ever reports a row whose price differs from the previous capture, and every such row has a price_change event beside it, so it now reads only the listings that recorded one in the window, looks up only the two captures behind each price move, and reads twelve hours rather than 36 (three migrations, the last the_capture_order_check_skips_price_changes_proved_false): 104s became 8s. The suite went green at 07:09 UTC on 2026-09-10.

The end-to-end sweep of all 1,000 enabled dealer sites, 2026-09-10. Every one has live listings. Nine had no complete traversal in three days: four easylist-hosted yards (jaxwholesalecars, assuredcars, ozcorpmotors, ramzcarsales) whose shared TLS certificate expired 2026-09-06 21:15 UTC --- verified with openssl, left enabled, nothing bypassed, resumes on renewal; three BMW sites (Geelong, Hobart, Mornington) with the Sydney BMW fault, a sitemap child answering 500, re-pointed to their stock sitemaps; Burnie LDV, whose /vehicles has been 404 since 2026-08-12 and which advertises no used cars, re-pointed to its used-car search so traversals complete. Dutton One (API ingest, no crawl_run rows by design) last observed 2026-09-09. Forty-three sites extracted under half of what their index listed; in every case checked the shortfall is the per-run detail budget rotating over a large yard (Tony White Group 2,015 of 8,389 found, budget 2,000) or a sitemap listing non-car pages, not a parser failure: the null-extract counts on those runs are 0 to 5. Two small sites parse poorly (ballinatoyota 4 of 25, bendigoisuzuute 4 of 12) and are noted for a look, not urgent.

The three portal yards. Westside Auto Wholesale 1,988 live, complete pass 2026-09-10 04:10; Scarboro Toyota 396 live, complete 06:20; Lexus of Perth 43 live, complete 05:28. All three are priority sources.

57The dealer page answered 500 for twenty minutes twice a day

What. Vercel raised a medium-severity anomaly on 2026-09-10: 176 failed requests on /dealer/[slug] in the five minutes from 12:35 UTC, against a daily average of one. The window is the tell. depth-boards runs at 02:25 and 12:25 UTC for about twenty minutes, and since 0294 its first statement took an ACCESS EXCLUSIVE lock on every table it truncates, live_snapshot among them, for the whole run. live_snapshot is what dealer_profile and dealer_inventory read; sale_speed reads sale_speed_board, locked the same way. A render inside the window waited the authenticator's eight-second lock_timeout, errored, and the page threw. A crawler walking the dealer pages at 12:35 turned a standing hazard into an alert; the same hazard had stood every day since 0294, unnoticed because nothing had asked for many dealer pages inside the window.

Fix. refresh_live_snapshot deletes and re-inserts instead of truncating, so under MVCC every reader stays on the old snapshot until the new one commits and no reader conflicts with the writer; the job's up-front lock list drops live_snapshot, and check_depth_boards_lock_order still reads clean (the_live_snapshot_is_replaced_under_readers_not_locked_against_them). The deadlock 0294 closed cannot reopen: the job no longer waits for anything while holding live_snapshot, and no reader waits on it. The dealer page's market comparison (sale_speed) is now read tolerantly and degrades to absent while its board is mid-rebuild; the yard's own figures never depended on a board.