Every change, in the words it was made with

Measure where the seller's listing age actually comes from

11:38 pm AWST · 80163fd

The comment said 79% of the corpus carries days_old. Two independent samples say 36.3% and 38.9%, and the breakdown explains both numbers: dealer-studio 97.4%, every other platform 0.0% — jsonld, carsguide, imotor, easycars, cox-radius and nextjs-embedded alike. The figure was never a parser coverage number, it was one platform's share of the corpus wearing a parser's clothes, and it moves when the source mix moves.

That matters because days_old is the only thing standing between days-to-turn and our own observation window, which averages 0.27 days against a dealer-published median of 55.

probe-listing-age.ts reads the retained bodies rather than guessing: of the ten largest jsonld sources, eight simply do not put the age on the page. So most of the gap is a publishing gap and cannot be parsed away — worth knowing before anyone spends a week on extractors for it. Two exceptions are real defects and worth fixing: nextjs-embedded returns an empty field map on pages whose HTML carries days_old and price_changes_count outright (westsideauto, 1,965 live listings), and trivett embeds a true ISO createdAt that advertisedDays already knows how to age (1,202).

The sampling RPC is deliberate: a jsonb key test across 184k live listings does not fit the API's 8s statement timeout, and ranking sources does not need a full scan.

Assert the sale tape means what it says, and write down why it drifted

11:26 pm AWST · ef8dd97

The interesting part of the bug was not the eight wrong rows, it was that every surface expressible in SQL had been fixed and every surface reading through PostgREST had not. The guard is a correlated subquery. The client cannot write one, so the fix could not follow the definition out to the places that needed it, and the drift tracked the query language rather than anything a reviewer would look for.

screen — a real car at a real price that nobody bought — and DATA-QUALITY 26, which also records the sharper lesson from days-to-turn. Its global median is 34 days with the guard or without it; one aggregate check would have passed while the per-model numbers it actually serves moved by up to 69 days. A bias that cancels in the aggregate is not small, it is one you cannot see at the altitude you are checking from.

Sold is a fact about a car, not about one listing of it

11:22 pm AWST · 689197a

The pulse (0135) and the market boards (0144) already agreed on what a sale still live. Two surfaces never got the rule.

The home page sale tape read delisted listings straight out of listing_current. Its own doc comment called them "the latest cars confirmed sold"; of the fourteen rows it was serving, eight were cars still openly for sale at another yard at the same price. A dealer syndicates one car, the marketplace copy drops out of that index, and the tape announced a sale while the dealer's own page still had it listed.

Valuation's days-to-turn had neither rule, so a car syndicated to three sites and dropped from one counted as a sale, and a car dropped from all three counted as three. Globally that hides — the median is 34 days with the guard or without it — but per model, which is how the valuation asks, it does not: Mustang 175 -> 106 days, Silverado 69 -> 105, X1 39 -> 72, GWM Ute 11 -> 26. Both directions, and large.

Neither guard is expressible in PostgREST, which is why both surfaces read the table directly and drifted. Both now go through an RPC. Aggregating days-to-turn server-side also lifts a silent 1,000-row cap that had been truncating common makes' cohorts with no error to show for it.

sibling listing still live. 65% of vehicles now carry a VIN, which is what makes the guard possible at all.

Discovery runs on the box, weekly, unasked

10:52 pm AWST · 031b986

It only ever ran when somebody remembered. The newest source was seeded 2026-08-14 and the corpus then sat at 184,000 live — not because the market stopped moving but because nobody asked it for more dealers.

Two steps, neither of which seeds anything. Mining reads links out of retained pages and costs no fetches; probing asks the top unprobed candidates for robots.txt and an inventory page and records a verdict against each host. Seeding stays a decision made after reading a probe — what this removes is the remembering, not the judgement.

probe-candidates gains --record, and it files failures as deliberately as successes. Five hosts probed on 2026-08-15 block the honest UA and will block it next week too; asking them again is wasted work and impolite. The failures are also the more useful half, because they say whether a channel is worth running again — and this sweep's said it is not: sixteen probed, one usable, against 60% on this method's first pass.

Weekly and nice'd, Sunday between the deep passes, three-hour ceiling.

Discovery candidates get a table, not a scroll buffer

10:44 pm AWST · 77de0b6

The corpus sat at 184,000 live and the newest source was seeded on 2026-08-14, not because the market stopped moving but because nobody asked it for more dealers. Every sweep so far existed as terminal output plus a hand-written seed script, so discovery happened exactly as often as someone remembered it.

mine-linked-dealers now records what it finds. Ranking is kept — mentions, and how many distinct sites linked to a host — so a domain three of our dealers point at outranks one that appeared once in a footer.

Seeding stays a decision. A host answering 200 has repeatedly turned out not to be a usable source, and every seed list here was written after reading a probe. This fills the queue; it never creates a source.

Valuation: a chassis code is not a trim, and an unknown odometer is not a match

10:34 pm AWST · 949565f

The C43 AMG was still wrong at $44,537 because only two cars carry that badge in 2019 — below the floor of three. There were three. The corpus files the car as "C43 AMG" twice and "C43 AMG W205 Four Wheel Drive" once, and W205 is the platform, not the trim. It now returns $54,990, against live 2019 C43s asking $58,439 and $54,800.

So trim identity gets its own key. normaliseBadge lowercases and stops there deliberately — it refuses to reduce "Sport GLS QF" to "GLS" — which is right for storage and wrong for matching. sameTrimKey drops chassis codes and drivetrain words, but never the leading token: "c200" is itself a letter and three digits, and a rule that stripped codes anywhere would erase it and reduce both "c63 amg" and "c43 amg" to "amg", valuing a C43 as a C63. Verified against GT vs GT Line, Sport GLS QF, M340i, RS6 and Wildtrak, all unchanged.

That fix regressed the C300, which is the useful part of this commit. Merging "C300 W205" in was correct — they are C300s — but those four listings publish no odometer, so the slope fit skips them while the weighted median took their raw $32,990 at full weight against five cars at the subject's own 60,000km asking $43,990 and $44,950. The answer dropped to $32,990 for a car worth about $44,000.

A comparable that cannot be corrected onto the subject is a weaker comparable — the same principle as the badge-shift cap. An ask with no published odometer now weighs half. Halved rather than dropped: it is still an ask, and on a thin model it may be most of what exists, but it is evidence about the model rather than about a car at this mileage.

2019 C-Class at 60,000km, all three now right: C63 AMG S $119,999, C43 AMG $54,990, C300 $43,990, C200 $36,073. Sanity suite unchanged, twelve cars within 5%, Ranger and RAV4 within 0.1% of 0.8.0.

Valuation 0.9.0: an AMG is not a C300, however many C300s there are

10:27 pm AWST · 20cb627

A 2019 C63 AMG S at 60,000km returned $44,032 — within $42 of the C300 median — against three live C63s at $97,990, $119,999 and $130,465. It now returns $110,262.

Three failures stacked. Only three cars carry that badge in 2019, and the trim restriction needs eight, so the set fell back to every 2019 C-Class, which is overwhelmingly C200 at $35,990 and C300 at $43,990. The badge-level term cannot rescue that: it shifts a comparable onto the subject's trim by their measured difference, but the shift is capped at 45% of the comparable's ask, so a $35,990 C200 reaches $52,186 and needs to reach $120,000. Twenty cars that cannot be corrected outvote three that need no correction.

So a trim priced apart from its model now restricts on three cars rather than eight. The threshold's reasoning — that a wider set beats a thin one — holds only while trims are priced alike; when they are not, a wider set is not a smaller error, it is a different car. 1.4x apart is well beyond trim drift and well inside the 3.3x here.

And when both restrictions cannot hold, which one gives way now depends on which error is worse. Only three cars are both 2019 and a C63, under the floor of four, so the guard was dropping the trim and keeping the year — letting 2019 C200s answer for an AMG. A 2018 C63 is a far better comparable for a 2019 C63 than a 2019 C300 is, so for a divergent trim the year gives way instead.

Tightness can also veto the label now, rather than only voting. The three subscores are weighted and summed, so sample 1.00 and recency 1.00 carried tightness 0.00 to 0.70 and printed MEDIUM CONFIDENCE beside a number wrong by $66,000. Sample and recency say how much evidence there is; tightness says whether it agrees. No amount of disagreeing evidence is a confident answer, so a set that dispersed caps at "low".

Sanity suite unchanged: twelve common cars all within 5%, Ranger Wildtrak, RAV4 GXL, C200 and C300 all identical to 0.8.0.

Still wrong and now at least honest about it: a 2019 C43 AMG returns $44,537 because only two carry that badge, under the floor of three. It reports "low" rather than "medium" — the answer is no better, but it no longer claims to be.

The cross-view check was re-verifying all of history, every run

09:58 pm AWST · 6fabaca

You saw the homepage with no figures in its hero and two panels reading "briefly unavailable". Postgres logs give the cause: bursts of "canceling statement due to statement timeout", beside repeated calls to check_cross_view_price_changes() taking 16, 17, 22 seconds.

It re-checked every price_change event ever recorded — 16,382 of them, each costing two lateral lookups into 1.18 million observations — on every run of the suite, which I ran many times today. The cost grows with both tables and has no ceiling.

It was also unnecessary work. Observations and events are append-only, so a price change verified last week cannot become wrong. Re-reading it proves nothing that was not already proved.

So it joins price_equals_identifier on the pattern this codebase already had for expensive checks: pg_cron computes it under a 15-minute budget, the request path reads the stored answer, and the reader returns computed_at so an answer over 12 hours old fails rather than passes.

The site was never broken. tolerant() degraded the panels instead of 500ing and recovered on its own, which is the behaviour it was built for — but it was degrading because the database was saturated, and that is worth fixing rather than admiring.

The crawler is no longer a laptop

09:50 pm AWST · 7aa0f46

spotlot-crawler, SYD1, 2 GB. 2.2 ms to Supabase against the laptop's 6.3 ms, so the box is three times closer to the database than the machine it replaced.

Verified before the cron came out: crawl, recanonicalise, enrich, insights and 14/14 integrity, the last filing a row that names its host — the cutover is a fact in the data rather than something to remember.

Records the two near-misses, which share a lesson. --omit=dev would have left the box unable to run anything, since tsx is a devDependency and the crawler runs TypeScript directly. And the first timer firing died with 226/NAMESPACE because ReadWritePaths names a gitignored directory that systemd needs before ExecStart. Running the crawl by hand had worked perfectly, because that path never touches systemd: test the mechanism, not the command.

FuelWatch moves to the box too

09:31 pm AWST · 615bcfc

econ_series takes WA daily fuel medians from a feed that publishes the current day only, so a day missed is a day that cannot be fetched later — the same class of asset as the observation log, and it was sitting on a laptop cron at 09:23 that macOS never fires after a sleep.

Persistent=true, so a day missed while the machine was down runs as soon as it is back.

The logs directory must exist before systemd, not during

09:18 pm AWST · 21ba2d5

The fast timer fired and the service died with 226/NAMESPACE: ReadWritePaths names /opt/spotlot/web/logs under ProtectSystem=strict, and systemd builds the mount namespace before ExecStart runs. logs/ is gitignored so it is absent from a fresh clone, and run-chain.sh's mkdir -p happens far too late — the script never ran at all.

Running the crawl by hand worked, because that path bypasses systemd entirely. That is the whole argument for testing the timer rather than the command, and for not removing the laptop's cron until the box has proved itself through the mechanism that will actually run it.

Pre-flight: the box would not have had tsx

08:48 pm AWST · ec8ff83

vps-setup.sh installed with --omit=dev, and tsx and typescript are both devDependencies. The crawler runs TypeScript directly through npx tsx, so the box would either have failed on its first pass or silently re-downloaded an unpinned tsx on every run. "dev" describes where a package sits in package.json, not whether this deployment needs it.

Installs everything now, and asserts `npx tsx --version` runs before going any further — a setup script that reports success while leaving the machine unable to execute a single script is worse than one that fails. Same fix in run-chain.sh, which reinstalls when the lockfile moves.

Caught by reading package.json rather than by an hour on a live box.

Reports say when their weeks close in Australian clock terms

08:29 pm AWST · 2359579

The UTC-week basis was already stated; now it is translated. Both the weekly and monthly period lines carry the eastern-standard-time close, with the daylight-saving caveat, because an Australian reader assumes a Monday-to-Sunday local week and the ten-hour offset deserves a sentence rather than an inference.

Also on the record in OPERATIONS.md: EOFY (30 June) tracking is agreed standing intent. First observed EOFY is June 2027; the June and July monthlies that year should carry an EOFY section cut by the Australian calendar from raw event timestamps, which the UTC recording spine fully preserves. Each later year compares against the last, which is the point.

Ops: nonces defeat dedupe on some sites, and it is not worth fixing yet

08:22 pm AWST · 156955c

An unchanged cheapcarstownsville page uploads six times because a lazy-load widget mints a fresh DOM id per render — same byte length, different sha. Normalising those ids before hashing is the obvious fix.

Measured first: of 60 disagreeing capture pairs, 17 collapse, all from one source, about 3 MB. A few percent against 180 GB a month, in exchange for the stored body becoming a representative rather than the exact bytes fetched — a poor trade in an archive whose point is being re-readable.

Recording the trap too: the ids are glued to a word, so a regex anchored on \b matches nothing and reports a confident 0%. The first run of this measurement declared the hypothesis dead when it had never tested it.

Ops: size the VPS against what the crawler actually moves

08:17 pm AWST · eeccf6a

74 GB a day inbound and 6 GB out, which differ by twelve times because bodies are gzipped before upload — 109 GB of HTML is 18.5 GB stored. Only egress is metered, so 177 GB/month is the number to size against rather than the 2.2 TB that looks alarming.

Records why Sydney is not negotiable (Supabase is ap-southeast-2 and the crawler makes several round trips per page, so the cheapest VPS on the market is in Europe and would be slower than the laptop), and that Australian bandwidth allowances are often cut to a fraction of the headline figure — the regional number is the one that matters.

Also records the storage trajectory: about 180 GB a month against the 100 GB Pro includes, free for five months, then roughly $0.02/GB. Worth watching rather than being surprised by.

The cross-cut earns its panel, and a caption stops overclaiming

07:46 pm AWST · ac87b8e

The state-and-fuel capture's third and last chart: market.ev_state_share draws Electric's share of stated-powertrain cars per state against the same day a week earlier, on the cross-cut table's own trailing-run clock, single rung at eight days - the table carries no event columns by design, so there are no flows to earn. Shares print to one decimal so states with nearby shares do not print alike: the first captured day has VIC at 9.3% against WA's 5.2%, with QLD, NSW, SA and TAS all inside a point of 7%.

Its adversarial review confirmed six findings, refuted one, all six fixed before deploy. The load-bearing one was a sentence this commit also removes from the state movement panel that shipped it first: "the not stated row holds cars whose listings never said a state" is false for 41% of that row's cars and 72% of the EV panel's, because state is a listing-level attribute and a syndicated car with one stated and one stateless listing sits in both rows, by design. The captions now state the membership rule instead of a purity the data never had. The fuel panel's twin sentence stays: fuel lives on the vehicle, and its "never" is true.

The rest: all three gates now handle the promotion-pending window - an evaluation that passed everything stores no block figures, and the old copy would have said "the days simply have to happen" for the day the ladder spends confirming its second measurement after every day already happened; the one-decimal rationale no longer bakes today's four-point spread into permanent copy; and this panel's gate prints a countdown only when the ladder actually supplied one, since its payload carries no day count to fall back on - the smoke test caught it asserting "0 of the 8" off a stale sixty-second cache.

Everything verified: 0148 byte-identical to the ledger, spec check clean, the migration's refresh_ladder() advanced no sibling streaks (0147 holding under exactly the condition it was built for), the ready view proven against real one-day data, prod serving all three gates.

Health check reads the integrity verdict, and notices its silence

06:17 pm AWST · 1b95967

The checks are the guarantee that no number on screen is invented, and health.ts — the one thing built for unattended running — did not look at them. It now alerts on a failing check, naming which.

The second alert matters more and is a heartbeat rather than a verdict. The chain runs crawl, recanonicalise, enrich, insights, then the checks, so if the crawler dies on a box nobody is watching, no row appears at all. Silence is the failure mode: a suite that never runs raises nothing, and a suite that raises nothing looks exactly like a healthy one. Twenty-four hours without a run is the alert, against a twelve-hourly chain.

Confirmed working across machines while adding it: the latest verdict on record came from host runnervmzvulz, a GitHub Actions runner, alongside one from this laptop. Both 14 checks, both clean. That is the host column earning its place — when the VPS takes over, the cutover shows up in the data rather than having to be remembered.

DATA-QUALITY 25: the corpus could not accept an improvement

05:15 pm AWST · 81bba5d

Retention was half a mechanism. The write path would not take a better attributes written once and never revisited, and the fill sitting on a path a re-observation never takes. Each hid the one beneath it, and all three were silent — the extractor improves, the tests pass, the numbers do not move.

Freshness is per chart, and a zero is not a missing row

04:50 pm AWST · ba4ae00

The fuel movement panel (0145/0146, swept into the crawl session's commits from the shared tree) went through the same adversarial review as its state sibling: ten findings confirmed, none refuted, all fixed before deploy. Migration 0147 and this diff are those fixes.

The one that mattered most: 0144 made ladder streaks advance only on fresh evidence, but scoped freshness to the cohort - and with two charts now sharing the market cohort, re-seeding one chart's measure stamped the other chart's streak forward, the single-measurement- counted-twice defect back through a side door. Both 0145 and 0146 had already done it to the state chart. evidence_at is now the stamp of each chart's own gating measures, the inflated streaks are repaired, and the seed-then-refresh migration pattern is finally as safe as its comments claimed.

The fuel RPC also learned what the state one never had to know: its table's row domain is data-dependent, so a powertrain reaching zero vanished instead of printing the strongest movement a demand panel exists to show, and a fuel absent on the level day silently lost its week of flows. The rows now come from the union of both level days and the flow windows; absence from a recorded day prints the zero it means, and a missing day still reads as withheld.

On the page: fmtShare prints a true zero as 0% instead of claiming a sub-percent presence; the caption no longer points at a cross-section panel that can independently soft-fail off the page; both movement captions say the newest day's tallies are provisional until the 00:10 finalisation; the fuel flows state their wider pool (every final exit, any condition, dated or not) against the cross-section's all-time dated used-and-demo Sold; and the cross-section now names its own window as Spotlot's whole watch. OPERATIONS.md loses a stale pre-0144 definition that contradicted the trailing-run semantics stated a paragraph later.

Genie builds scissor lifts, not cars

04:45 pm AWST · 94014ec

A 2023 GS 2632 reached the review queue from a dealer that sells plant beside its stock. Marked out of scope rather than left pending, so the queue keeps meaning "a human needs to decide this".

--max-detail: exercise the delist check without re-crawling a yard

04:34 pm AWST · fd23ac8

Westside is 1,750 detail pages at a four-second crawl delay, which is 2h43m, and asking one dealer for that twice in a day to verify a code path is not a reasonable thing to do to them.

The override lowers a single run's detail budget and touches no configuration. It does not weaken the delisting: `complete` describes the index traversal, and a listing still in the index but past the budget is protected by presentUrls rather than treated as vanished — the same reason a source's ordinary budget does not make its traversal partial. A smaller budget refreshes fewer prices; it does not make a delisting less certain.

Two VPS faults found by measuring the pass, not guessing at it

04:31 pm AWST · 97b522f

A full traversal of 700 sources ran 6h05m and was still going. The deep service capped TimeoutStartSec at 6h, so systemd would have SIGTERM'd every pass a few sources short of the end, permanently. Raised to 10h, which kills a hung pass and not a slow one. The signal path itself is complete=false, so a half-finished traversal cannot infer a delisting.

The worse one was the cadence. run-chain.sh refuses to start a crawl while another runs — correct, because per-host politeness is enforced per process and two crawlers on one dealer is how that host answered 503 to both. But a six-hour pass on a six-hour timer is always running, so the two-hourly fast pass would essentially never have fired, and the fast pass is the one that feeds price movement. Deep is now twelve-hourly, which leaves real gaps for it.

Also the one client-visible bug class this started from: four "use client" components in the whole app, and only Freshness touched a clock. The other three initialise state from props and cannot diverge. Fixed the single number in the app formatted without an explicit locale (market/page.tsx), which is server-rendered so it never mismatched, but its output depended on the host's LANG.

"29779697m ago" was the epoch, served by a cold cache

04:20 pm AWST · f40d7b9

The homepage reported its figures as computed in 1970. The data was fine — $117.5M, 4,539 cars, all correct — but the age beside it was 1,786,000,000 seconds, which is Date.now() minus zero.

cached() writes a holding entry before a cold compute with `computedAt: hit?.computedAt ?? 0`, and that branch is only reached when there is no previous value, so the 0 is by construction. The first caller returns `new Date()` and is right. A *second* caller arriving while that compute runs awaited the same promise and returned `new Date(hit.computedAt)` — the placeholder, still zero. It now reads the timestamp back after the compute lands.

It hid for so long because the server cannot see it. Freshness starts `now` at `computedAt`, so every server render shows an age of zero whatever the value is, and the truth only appears when the browser hydrates and substitutes its own clock. Cold process plus two concurrent

The component now refuses to print an implausible age at all, and says "age unknown" instead. The cause is fixed; this is the seatbelt, because the failure mode is silent on the server and loud in front of whoever is looking at it.

A day is only final after it ends, and other review findings

04:03 pm AWST · ed83963

The per-state movement panel (0143, swept into cf31842 with the ladder and lib halves) went through adversarial review before deploy: thirteen findings confirmed, one refuted, all thirteen fixed. Migration 0144 and this page diff are those fixes.

The one that mattered most: every rollup day's event columns froze at the 15:55 UTC write, so 15:55 to midnight never entered any day's row, and the movement panel's flows would have run 10-30% below the pulse and the weekly report while presenting the same car-based definition. finalize_daily_events now rebuilds the event-derived columns of each closed UTC day at 00:10 from the append-only tables (stock columns keep their only-the-current-snapshot guard), and finalising 2026-08-14 lifted its sold count from 1,474 to 1,560 cars - the truncation made visible. The ladder itself gained a rule it turned out to need: streaks only advance on fresh evidence (ladder_state.evidence_at), because the bare refresh_ladder() call in 0143 counted one measurement twice and early-promoted four model cohorts past the two-confirmation damper. state_days_contiguous became the run ending at the latest day, so a stalled capture demotes instead of coasting on old glory.

On the page: the gate clamps its countdown and a read failure at an unlocked rung says so instead of claiming the days have not happened; the levels caption says calendar days like the SQL means; a rung-2 flows withholding names the mid-window hole rather than a requirement already met; the drift message ceilings to one decimal so it cannot print the threshold it claims to exceed; "sold and repriced" became "cars sold and prices cut" (the terminal owns "repriced" for cuts plus rises); the car-counts-once caveat is scoped per row the way 0140 declared it. And the panel above stopped being the one unlabelled unit on the page: market_state_board counts cars now - it is truncated and rebuilt six-hourly, so no recorded series changed unit - and "Live cars by state" finally agrees with the movement table in both unit and ordering, where QLD 48,649 listings sat over 26,696 cars with nothing saying why.

Vehicle specs stop being frozen at first sight

03:51 pm AWST · cf31842

Three layers of the same fault, each hiding the one beneath it.

Re-parsing recovered prices and no conditions, because condition is written on insert as `?? "unknown"` and the blanks-only fill did not cover it — a sentinel is not a null. Fixed, and then re-parsing recovered conditions and no body types.

Because body type, transmission, drivetrain and fuel live on `vehicle`, and resolveVehicle finds a row by fingerprint and returns it untouched. So every vehicle attribute was written once, at first sight, and never looked at again. Today's three extractor fixes could not reach a single existing of the corpus, improving only for vehicles inserted afterwards.

And fixing that alone still did nothing, because resolveVehicle is only called when a *listing* is created. A re-observation never reaches it. The fill had to go on the path a re-observation actually takes.

Blanks only, and identity excluded. make, model, badge and year decide which cohort a car is valued against and moving one silently would change an answer already given; those belong to the taxonomy sweep, which is deliberate and reviewable. What is filled is description — a null becoming "wagon" says nothing different about which car this is.

cox-radius also now takes the spec fields from the page's JSON-LD, which carries them in full (Sedan / Sports Automatic Dual Clutch / Petrol - Premium ULP / RWD) while the DOM harvest refuses them to the same uniqueness guard that refuses the price.

xtremecarsales, from retained bodies: 19 of 33 live listings gained body, transmission, fuel and drivetrain, exactly the 19 with a retained capture. All 14 integrity checks pass.

Re-parse retained bodies, and the two things that stopped it working

03:12 pm AWST · a008285

The payoff for keeping every distinct body is that a better parser can re-read the past without re-crawling. Today's three extractor fixes only reached future crawls; this reaches the corpus already held.

reparse.ts needed three corrections before it could be trusted at scale. It took the newest N captures, which re-read one busy car five times and never reached the quiet one — now deduplicated by URL, newest kept. It was unpaged, so PostgREST's 1,000-row cap silently truncated it while it reported success. And it wrote inline listings with the default source view of `detail`, which mislabels a search-index price as a listing-page price — the exact confusion that once manufactured six price changes no dealer made, and one check_cross_view_price_changes cannot catch, because the mislabel removes the boundary it looks for.

It also had to learn not to raise the dead. observeListing relists a delisted listing when a detail view shows the car again, correctly, because the car's own page is what may bring it back. Feeding it a capture from last Tuesday would resurrect every car sold since — bdkauto alone has 96 confirmed sold whose pages still say so. Detail re-parses are now filtered to listings that are still live.

Then the re-parse recovered every price and no conditions at all, which exposed the real gap: condition is written on insert as `?? "unknown"` and was never revisited. The fill-in block already exists for exactly this — "a listing first seen by a weaker parser kept its gaps forever" — and condition was missing from it, because "unknown" is a sentinel rather than a null and the blanks-only rule read it as a value already recorded. It is a blank wearing a value's clothes. Still blanks only: a listing that says "used" is not overwritten by a later reading.

Four cox-radius sources, from retained bodies with no fetching: toowoombagmsv 7 priced -> 128 of 128 and 0 -> 121 with a condition, xtremecarsales 14 -> 31 and 0 -> 19, bdkauto 3 -> 18 and 0 -> 29, selectautossa 0 -> 4 and 0 -> 35. Zero events written across all four — no price changes, no relists, nothing invented.

The bare-Product sites, and the one field worth taking from the name

02:17 pm AWST · c41f736

Continuing the empty-field sweep found a group rather than a defect. 53 jsonld sources holding 10,225 listings never yield body type, transmission or fuel — the same 53 for all three — nor condition nor odometer, while every one yields a price. Their JSON-LD is a bare Product: name, brand, image, offers. No Car or Vehicle node, so those properties are absent rather than unread, and no parser change reaches what is not there.

One field was recoverable. cars4us names its products "Used 2019 Ford Focus Trend For Sale ID69213", so condition falls back to the word the name opens with — required to sit immediately before the model year, because this runs on all 367 jsonld sites and an opening word alone would read "New Shape Ranger" as a new car.

Honest about the yield: 1 of the 14 largest such sources, about 920 listings. Verified 4/4 on cars4us, and autopact — which reads itemCondition properly — unchanged at 4/4 with a mix of used and new, so the fallback only fills silence.

The rest is documented as a ceiling rather than a todo. The odometer is in the page text on some of these sites, and taking it means the ambiguous-DOM problem cox-radius already showed the cost of. These 10,225 are priced stock with make, model, year and badge: real coverage, thin comparables.

Read schema.org's drive-wheel URIs, not just the plain word

01:48 pm AWST · d8119d3

jsonld's drivetrain null rate is 53.9%, which looks like variation across sites and is not — it is bimodal. 127 sources holding 23,958 listings sit at exactly 100%, and 102 holding 18,080 read it on more than 90%.

Doncaster BMW publishes "driveWheelConfiguration":"https://schema.org/AllWheelDriveConfiguration", which is entirely correct schema.org. The extractor already read that property, passed the URL through as a string, and it failed drivetrain canonicalisation and landed as null — so the field read as absent on a site stating it plainly.

The enumeration has four members and they are mapped rather than pattern-matched, because FrontWheelDriveConfiguration and FourWheelDriveConfiguration both begin with F and a loose match would quietly turn a front-wheel-drive hatch into a four-wheel drive. The object form with a name or @id is handled too. Sites publishing the plain word are untouched.

Measured on retained captures across the twelve largest never-drivetrain five BMW dealers go from nothing to both pages. The other five publish no drivetrain in their JSON-LD at all and correctly still return none.

DATA-QUALITY 24: a field at exactly 100% empty is a question

01:44 pm AWST · 30e4eca

Records the method behind today's two extractor fixes, and the one it cleared. Null rates per field per platform, with nextjs-embedded as the control at 0.5-2.3% across every field. Exactly 100.0% is the signature, because a field the parser sometimes misses lands somewhere untidy while a field no code reads lands on a round number.

Includes the negative result, which is the part worth keeping: easycars is at 100.0% for drivetrain and its extractor has no drivetrain logic, so the signature was perfect — and its pages carry no drivetrain text in any form, so nothing is wrong. The prize is a field the source publishes, not one we would like to have.

Body type for 19,221 autotrader listings, from the URL that always said it

01:16 pm AWST · bb27fb6

Two sites share this extractor and only one of them was yielding a body 19,221. The cause is in the spec table, which flattens differently there and delivers the cell as navigation text — "s Convertibles Dual cab utes Hatchbacks People movers Statio". shortSpec refused it, correctly. Having no second source for a field the URL states outright was the mistake.

Both sites put it in the last path segment, and this file already parses those segments and already documents the shape: /car/{id}/{make}/{model}/{state}/{suburb}/{body}. So the fallback is the segment, guarded on there being exactly seven of them — because this file also carries the scar of trusting a segment whose position was assumed, when a URL that did not line up put "SUV" in the state column.

Verified on retained captures, no fetching: autotrader 8 of 8 with a body type against 0 before, carsguide unchanged at 8 of 8 and still reading its own spec table, so the fallback only fills silence.

Noted while here and not addressed: body_type is not canonicalised across the corpus. "hatchback" appears 8,302 times and "hatch" 4,725, alongside 5d hatchback, sportshatch, 5 door hatch and four more. This change adds the spelling that is already the largest bucket, so it joins the fragmentation rather than widening it, and the segment views read through the enrichment layer rather than this column.

The coverage gap is 19,263, not 25,046 — the rest is merged duplicates

12:46 pm AWST · b068aad

source_max_found() reports disabled and merged sources too, and a merged storefront's found-count is cars we do hold, under the yard they belong to. The tell is identical pairs: cherymoorooka and cheryspringwood both find 718 and hold 0, bartonsgwmhaval and bartonscapalabagwm both find 274 and hold 0. Franchise storefronts over one inventory system, all already carrying merged_into_source_id, none a coverage gap. 5,783 of the total was that.

Of the real 19,263, autotrader is 7,393 and structural: 4.5 seconds a page against a 3,000-page budget is already 3.75 hours, so the four-hour rule binds before the budget does, and it publishes no rate directive that would justify going faster. The rotation reaches the remainder across passes rather than losing it.

Also corrects the read on state skew. NSW having the most sources and fewest listings each looks like a capture failure and is the opposite: NSW is 5.6% short of what its sitemaps advertise, the best of the large states, against WA at 19.2% and VIC at 10.2%. So NSW being 21% of the corpus while being the largest vehicle market is a question about which yards we hold, not about how much of them we read.

cox-radius: 1,437 cars we were counting and knew nothing about

12:21 pm AWST · 4a30ef4

Chasing "14% of live stock has no odometer" found something better. Most of that 14% is new and demo cars, where there is nothing to read — the real gap is about 1,609 used cars, under 1% of the corpus. But the breakdown exposed cox-radius at 100% missing, and pulling that thread: 1,437 live listings across five sources, no price on 1,404 of them, no odometer on any, and condition "unknown" on all of them.

The cause is a guard doing its job too well. A radius detail page carries "recently viewed" cards for other cars, so the extractor promotes a field only when the page yields exactly one candidate. Correct in principle, and it fired on every page in practice.

The guard stays — guessing between two prices on a page is exactly the mistake it prevents — and the page's own JSON-LD now fills what it refuses to guess. offers.price belongs to the vehicle the page is about, not to a widget beside it, so it answers the question the DOM could not. Only fields left undefined are filled; make, model and slug stay with this extractor, which knows its URL shape and whose spellings the taxonomy has already been taught.

Measured on 20 retained captures across all five sources, no fetching: 11 priced and 20 with a condition, against 0 and 0 before. The header's claim that detail extraction was unverified is now false and says so.

Size the three "extractor not yet built" marketplaces; one was a duplicate

12:15 pm AWST · d604d3a

Their notes read as free coverage waiting on a little work. Probed: one of the three is already in the corpus — market:peterwarren is the same host as dealer:peterwarren, which is enabled on dealer-studio with 1,689 live listings, and is now marked merged so the yard cannot be seeded twice.

The other two are real but not cheap. Both answer 200 on their inventory pages and neither is recognised by any extractor, and scanHost finds no listing URLs on either — easyauto123 publishes no usable sitemap, suttons has one that lists no vehicles. So each needs a new extractor *and* pagination-based discovery, because the sitemap shortcut that made the dealer corpus cheap is not available. A discrete project with an unmeasured yield rather than an afternoon.

Written into robots_notes as well as the doc, so the next probe starts from the measurement instead of the optimistic note that preceded it.

Soft 404s become delistings, labelled as what they are

12:05 pm AWST · d01d820

cars can otherwise never show days-to-sell, because their dead listings answer 200 with no SOLD text and both accepted proofs miss.

This is the only delisting Spotlot infers. A 404 is the server saying the car is gone; a SOLD retitle is the seller saying it sold. Here the page says nothing and we conclude from its shape, so the difference is carried to the surface rather than smoothed away: events are marked confirmed_via 'soft 404', dealer_profile returns a basis per sale, and the page reads "N confirmed sold, M no longer listed" with a column saying which is which. Nothing that said "sold" changes meaning.

Four gates, all denominated in the harm. Three days absent, because sixty recently-vanished candidates gave zero soft 404s while twelve gone five days gave twelve. Positive page evidence — it offers other cars, or several dead listings share one document — never mere absence, which is precisely what a broken parser produces. Detail-loop parse rate at 95%+ over 20+ pages, measured on index-carried pages this check never touches. And a quarter of live stock as the ceiling, tighter than the sold sweep's half because that one carries the seller's words and this carries our inference.

The evidence ladder widens deliberately rather than loosening: a soft-404 delisting is admitted only with its basis and age stored, and a new check requires every one to carry the parse rate and either five other cars on the page or three listings sharing it. Verified by hand first — villagemotors serves its homepage, byte-identical at 253,284 bytes, for two different Havals, while a live car returns 389,890.

The partial-run rule is about absence, and the sweep does not use absence

10:43 am AWST · 85bf6e4

An integrity alert fired within the hour: 4 delistings from an incomplete crawl, all dealer:bathurstgwmhaval, all from the detail sweep shipped this morning. Every one is a genuine sale — the evidence stored with them is the dealer's own page title, "SOLD 2018 Subaru Impreza 2.0i-L G5 in White | Used Hatch | Stock #706087", and three more like it.

What broke is the check's premise, which the sweep invalidated. "A delisting may only come from a crawl that traversed the whole inventory" is right for one that rests on absence: not-in-the-index means gone only if you saw the whole index, and inferring otherwise once marked thirteen cars sold that were all still answering 200. The sweep never touches absence. It reads the seller's SOLD off the car's own page while the crawler is already holding that page, and a traversal stopping early cannot make that page say something else.

So the check now asks its real question rather than a proxy for it. This narrows the assertion, it does not weaken the guarantee: check_unconfirmed_delistings still demands evidence for every delisting without exception, whichever path produced it, and absence alone remains worthless on both. Suite is green on its own terms rather than because something was switched off.

Keep the feedback loop after the crawler leaves this laptop

10:34 am AWST · 3f66533

Two things would have quietly regressed on a VPS, and both are the part of the setup that finds bugs rather than the part that fetches pages.

The integrity suite printed its verdict and stored nothing. That was fine while the chain ran on the same machine the work happened on; on a box it goes to journald, so the checks could start failing and nothing outside an ssh session would say so. They now write one integrity_run row per run, with every check's verdict and the hostname that produced it — which also makes the cutover self-evidencing, since the host changes. The write never throws: a suite that passed but could not file its result must still exit zero.

And cron ran the working tree, so a fix was live on the next pass. That loop is how most of today's findings were made: a pass reports something odd, the cause gets fixed, the next pass says whether the fix worked. A box pinned to whatever was cloned breaks it silently — passes keep succeeding, against old code. run-chain.sh now takes main before each pass, --ff-only so local drift stops the update rather than being merged into something nobody wrote, and reinstalls when the lockfile moved. A failed pull is not fatal: the previous commit is a working crawler, and skipping a pass costs more than running slightly old code.

Turnkey VPS deploy: one script, systemd timers, and the cutover order

10:24 am AWST · 44680be

not need a credit card — the box itself needs an account and a card, so it stays with Taj.

vps-setup.sh is idempotent and refuses to enable the timers until .env.local has a service-role key. Sydney region, and the reason is Postgres rather than the dealers: Supabase is ap-southeast-2 and the crawler makes several round trips per page, so the cheapest VPS on the market is in Europe and would be slower than the laptop it replaces.

systemd rather than cron for one property — Persistent=true runs a timer missed while the machine was down. That is the entire failure being fixed; cron skips it silently and a skipped pass cannot be backfilled.

The fast pass goes two-hourly, up from six. Price changes come from frequency and an always-on box is what makes frequency affordable. The deep pass drops --minutes: that cap exists because Actions bills by the minute, and the four-hour-per-source rule still bounds any one source.

Runbook carries the cutover order, because the laptop crontab must come out only after the box is verified. Two crawlers on one dealer is DATA-QUALITY 18, and the in-flight guard cannot see across machines. Also records how to close run rows orphaned by a killed pass — 23 of them from one SIGPIPE today — and that they must close as complete = false.

The crawler is a laptop, and that is the biggest risk we carry

10:13 am AWST · acbf1f4

The 89%-in-24h freshness is real and is produced by a MacBook. A local crontab entry runs unbounded passes four times a day; GitHub Actions contributes about 42 minutes a day across four time-boxed passes, and crawl.yml already says what that alone buys — several days per sweep.

Measured 2026-08-14: 100-167 runs and 20,000-40,000 pages an hour for sixteen hours, then 1-2 runs and under 350 pages. macOS cron does not fire missed jobs after a sleep, so a closed lid is a skipped pass.

Written down because it outranks every other infrastructure question. Extractors, taxonomy, boards and parses are all reconstructible from retained bodies; observation history is not. A week with the lid shut is a permanent hole in every trend that spans it, and at 4.6 days old the ratio of what is at risk to what exists is as bad as it will ever be.

Also raises twelve detail budgets, +3,303 pages a pass, closing part of the 23,937 listings our own sources advertise and we do not hold. Safe against the Actions budget because those passes stop at their deadline.

Migrations for the RPCs the evidence work applied directly

09:40 am AWST · 84cda32

Four files the tree was missing: the two capture samplers, the no-sales-yet source list, and the drop of the probe RPC whose create is still in the database's migration history. Without the last one a rebuild from the tree would leave a function that was deliberately removed.

The soft-404 test is what the page offers, not what it lacks

09:21 am AWST · 9e3c137

162 sources holding 30,466 cars can never record a sale: their dead listings answer 200 with no SOLD text, so neither accepted proof fires. "The extractor found no car" was the obvious third proof and the wrong one — a broken extractor returns null for every page and would invent a sale for every car on the site.

Measured on Westside, the page says more than that. A dead listing there serves the stock index: 18 other cars and none of its own. A live listing returns its car plus 10 related. So the test is positive rather than absent — the page offers other cars while not offering this one, which is the site saying the car is gone without words. A broken parser finds no car and no others either, and fires nothing. That asymmetry is the guard.

Age is the second condition. Sixty recently-vanished candidates gave zero soft 404s because an index drops a car for a day and picks it back up, while twelve cars gone five days were twelve for twelve. Three days.

Counted only. The run now reports how many candidates meet the whole rule beside the parser's health on detail pages, so the next pass says what it would have done across 698 sources before it is allowed to do it.

Twelve of twelve, and the rates corrected to what was measured

09:06 am AWST · bf165cd

The soft-404 rate was the open question from DATA-QUALITY 21. Asking the right population answers it in twelve polite fetches rather than the 2h43m a Westside traversal costs: its twelve longest-vanished live listings are all soft 404s — 200, no marker, no car, titled "Used Cars For Sale in Perth". Zero provable today, zero still listed.

That sits beside the same source's delist check finding zero soft 404s among sixty candidates, and the two agree. The check's sixty were the recently vanished, held there by the id ordering, and those really are still for sale — an index drops a car briefly and picks it back up. Which candidates you ask decides what you learn, which is why the rotation had to land before the signal could be judged.

Still acted on by nothing. Twelve for twelve is one dealer, and a broken extractor would return null for every page.

Also corrects two rates to what was actually measured. The stratified figure is 2.3%, not 1.8% — that one came off a sample PostgREST had truncated at 1,000 rows, which stopped at dealers starting with b. The corpus figure stays 1.2% on a uniform draw, roughly 2,200 cars, and the "3,300" quoted earlier came from applying the stratified rate to the whole corpus, which is the error the two samples exist to avoid.

An override for dealers verified to keep sold cars, and paging for the harness

09:02 am AWST · aae95de

The sweep's ratio guard is a permanent block on exactly the dealers it was built for: BDK Automotive would retire 96 of 137, and from a ratio alone that is indistinguishable from a marker matching everything. So there is now a per-source flag, set only after someone has fetched one of that dealer's pages and read the sentence themselves. It still refuses above 90% — no dealer sells its whole yard between two passes.

Set for bdkauto, whose page says "This vehicle has been sold but we're happy to help you" under the title "All Stock". Not set for bendigomazda, which the sample put at 6 of 10: fetching two of its longest-unseen listings returned 200 with no marker at all under the generic title "New Mazda For Sale Bendigo". Those are soft 404s, not sold pages, so its behaviour is mixed and the ratio alone would have whitelisted it wrongly. It will trip and report instead, which is what the guard is for.

The harness was reading a truncated sample and calling it a corpus rate. PostgREST caps a response at 1,000 rows whatever the function returns, so --per-source 10 silently became "the first hundred sources" — the output stopped at dealers starting with b. Paged now. The uniform 1.2% figure is

Sunbeam is a car

08:40 am AWST · a28cbb9

A 1961 Alpine reached the review queue. Rootes Group marque, gone since 1976 — the roadster, not the later Talbot hatchback of the same name.

Among never-checked candidates, ask the longest-gone first

08:15 am AWST · f2e6d39

The round-robin sorts by delist_checked_at, and every listing starts null — so the tiebreak decided the first run after the fix, and a tiebreak on id reproduced exactly the order that starved the queue. Westside takes 2h43m a run; that is a whole run spent re-asking the same sixty before the rotation could begin.

Falling back to last_seen_at is the better question anyway: a car missing from the index for five days is likelier gone than one missing since this morning. The two Westside pages verified by hand as soft 404s were last seen on 10 August and had never been checked once.

DATA-QUALITY 22: the seller said SOLD and we filed it as stock

08:13 am AWST · 6fea994

Decide the sold sweep once, before writing anything

08:11 am AWST · b95e153

The first guard fired on the 127th marker of 252 pages and had already committed 96 delistings. Those happened to be right — BDK Automotive really does keep sold cars listed forever behind "This vehicle has been sold but we're happy to help you", under the generic title "All Stock" — but had the marker been broken they would have been 96 fabricated sales. A guard that stops after most of the damage is not a guard.

Sold pages are now collected during the traversal and decided together once it ends, so nothing is written until the whole picture is known.

The threshold also changed shape. Counting pages that carry a marker punishes exactly the dealers this was built for: one who never removes a sold listing shows a permanently high marker rate and would trip forever on cars already delisted. It now counts cars this run would newly retire against the cars the source holds live — a broken marker takes nearly the whole inventory at once, which is the thing worth refusing.

Corrects the corpus figure while here: 1.2% of live listings carry a marker on a uniform draw, roughly 2,200 cars. The 1.8% quoted earlier is a per-source stratified rate, which answers how many dealers rather than how much of the corpus, and reads high because it weights a forty-car dealer like a nine-thousand-car one.

The fuel panel says only what survived its own review

07:57 am AWST · 5a295d3

The state and fuel capture (0140), the /market powertrain panel (0141) and the review-findings migration (0142) landed in the tree through c710e79's sweep of the shared checkout; this diff is the page's share of those review findings. Eleven confirmed by adversarial verification, five refuted, all eleven fixed before deploy:

- The unknown-row footnote claimed the sources that omit fuel omit listing ages too. The corpus says the opposite (66.7% of unknown-fuel delistings carry an age against 50.6% for stated-fuel); the row is thin because its 17,028 cars are ~85% non-used stock. The footnote now states the observed numbers and nothing else. - "Nearly new" rendered on a null-check, not a value-check, and would have kept printing as the used-EV pool ages. The adjective is gone; the figures speak. - The sold pool's used-and-demo restriction is now stated, with the note that Live cars spans every condition so sold over live is not a turn rate. - FuelSlice.sold's doc comment claimed identity with the pulse that the data disproves (1,737 vs 3,358); it now names the actual shared invariant and the actual subset relation. - The lede and the share column now share one fmtShare rule, so a sub-half-percent fuel reads "<1%" in both places instead of an untrue 0% in one of them. - Ask and stock-profile cells gate at four cars like the turn cells, so a future one-car fuel cannot print its only price as a median.

The SQL side of the same findings is 0142: fuel cuts collapse the state dimension before counting (Petrol 1,312 recorded became 1,235 cars), write_daily_rollup refuses a p_day that is not the snapshot's own UTC day, the data-dependent domains clear the day before rewriting it, and turn_filtered/turn_segments count cars — /moving's "3,598 matching sales" (958 of them new stock its own copy said was excluded) is now 1,739, two cars off the boards built half an hour earlier and identical in definition.

Read the seller's SOLD where we already hold the page

07:49 am AWST · c710e79

soldMarker was called in exactly one place: the delist check's verify step, which only runs on a car that has left the index. A dealer whose index keeps the car was never asked at all — 31 sources holding 2,035 cars have never produced a single candidate.

Measured against retained captures, with no fetching, 18 of 977 live listings (1.8%) already carry a marker on their own page: "SOLD 2023 Mitsubishi Outlander ES ZM", "this vehicle has been sold". Across the corpus that is roughly 3,300 cars counted as for-sale stock, sitting in dealer counts, median asks and valuation comparables.

This is not a new inference. A SOLD retitle is the seller stating the outcome and is already accepted proof; it was simply never read on the path we fetch most. cheapcarstownsville: 32 of 88 pages carried one, 16 belonged to cars we held live and are now delisted, 16 to cars we never had — those are skipped rather than created, since a car first seen already sold has no asking-price history to attach to.

Two guards come with it. Only the car's own page may relist it, because observeListing resurrected any delisted car on sight and a sold car that stays in the index would flap its status every pass. And the sweep stops if more than half the pages found carry a marker, which reads as a broken marker rather than a sold-out dealer — the line sits well above honest outliers like the 36% above and far below the ~100% a storm would give.

Chunk the delist-queue stamp; 400 ids is 15 kB of URL

07:16 am AWST · a3a38b2

The batched update was written when the ceiling was 60 and its comment said so. Raising the ceiling to 400 in the same series made that stale: a PostgREST `in` filter is query string, and four hundred uuids is roughly 15 kB of request line, past Kong's limit. Chunked at 100, about 4 kB.

A dealer with no sales is told why, not told to wait

07:13 am AWST · 03f1c43

The zero-sales copy promised "these accumulate with every pass". For 162 dealers holding 30,466 cars that is false: their sites never confirm a sale, so the count stays at zero however long we watch. Westside Auto Wholesale is one of them, and it is the page the question came from.

Now it says none-yet is not none, names the two things that count as confirmation, and says plainly that some sites do neither and that we would rather show nothing than infer a sale we cannot prove.

Measure the parser's health on detail pages, not on the blend

07:11 am AWST · 875b205

listings_extracted is incremented once per inline listing as well as per detail page, and an inline listing was already parsed out of the index. So a source whose detail pages stopped parsing entirely can still report a ~100% parse rate, and 528 of 698 dealer sites have inline indexes. The blended figure is not extractor health and cannot guard anything.

The delist note now carries detail parse counts alongside the soft-404 count, which is uninterpretable without them: "N of them no longer describing a car" means a dealer's pages changed if the parser is fine that run, and means the parser broke if it is not.

Fetch failures stay out of the ratio — a 503 says nothing about the parser. Only pages that answered 200 and were handed to it count.

Runbook: why a dealer shows no sales

07:06 am AWST · 0c309df

Records the two faults and what neither of them fixes, with the queries to read a source's delist queue and the per-run soft-404 count.

Absence from a sampled index is not evidence of a sale

07:04 am AWST · e66131f

The delist check rests on one premise: a live listing absent from a complete traversal has genuinely disappeared. That is true of a dealer's sitemap and false of a marketplace's search results, and `complete` does not distinguish them — it only says we reached the end of the pages we ask for.

carsguide's index is a union of paginated buy-a-car queries capped at 1,500 pages. It returns a steady ~2,000 listings a run against 4,331 held live, so cars leave our view because the window moved. Its backlog grew to 2,268 and all 60 checked each run answered 200. Both enabled marketplaces have recorded no sale, ever, while 423 of 698 dealer sites record them on exactly this basis.

Left alone, the previous commit would have spent up to 400 fetches a run on the source where the premise is weakest. The check is now skipped there and says why on the run.

The default follows kind rather than per-source config: sixteen marketplaces are disabled, and enabling one later must not silently inherit the permissive answer.

A large delist backlog can claim a quarter of itself

07:01 am AWST · 70a59f9

Sixty checks a run is the right floor and the wrong ceiling. Rotating the queue makes a backlog drain, but market:carsguide holds 2,268 candidates the check has never reached — 38 runs before the first is asked twice, which is far too slow for days-to-sell to mean anything. It has 4,331 live listings and has never recorded a sale.

A backlog may now claim a quarter of itself, capped at 400, which drains carsguide in six runs and leaves every source under 240 candidates exactly where it was. Across the twelve starving sources this is about 1,158 extra fetches on the first pass, against roughly 180k in a full one.

The raised ceiling needs the pass deadline, which the delist loop never flight billed a ten-minute pass at nearly thirty. Stopping early only means fewer checks, and unchecked is now counted from what was actually asked rather than from the budget, so a short pass reports the backlog it really left.

The delist check takes turns instead of asking the same sixty

06:59 am AWST · 4771960

markDelistings pages live listings by id because that is the only way to page them, then checks candidates.slice(0, 60) of that same order. Both the order and the filter are stable, and a candidate that answers 200 forever stays a candidate forever, so the first sixty were a permanent wall.

Westside Auto Wholesale shows it end to end. Its backlog grew 3 -> 12 -> 21 -> 70 -> 97 -> 105 -> 150 over four days while exactly 60 were checked each run, and it has never recorded a single sale. Every car that genuinely sold arrived behind the wall and was never fetched once. 268 of 698 sources have no sales data at all.

Candidates now sort least-recently-checked first, never-checked before that, and every checked row is stamped whatever the outcome — including a failed fetch, which proves nothing about the car but would starve the queue just as effectively if it kept its place.

Nothing about what counts as proof changes. This decides which cars get asked, not what the answer means.

It is also a prerequisite for measuring the soft 404s: the starved prefix is made of cars that really are still listed, so the pages that no longer describe a car sit in the backlog the check never reached.

Count the soft 404s, decide nothing on them

06:56 am AWST · 2a6626b

268 of 698 sources have never recorded a sale, which is why so many dealers show no days-to-sell. Westside Auto Wholesale is the clean "Used Cars For Sale in Perth", and no SOLD text. Neither proof markDelistings accepts fires, so the car stays live forever. One pass left 1,401 such cars unresolved across 302 sources.

The page no longer describing a car looks like a third proof. It is also the most dangerous one available: a broken extractor returns null for every page and would invent a sale for every car on the site. So this measures the rate and changes no decision — markDelistings counts refused candidates that no longer describe a car and reports it on the run.

The first attempt to measure it from outside was wrong and is written up as DATA-QUALITY 21. It sampled candidates as live rows with a stale last_seen_at, which is not what the delist check uses: candidates come from presentUrls, and a car in the index but past the run's detail budget is protected deliberately while its last_seen_at goes stale. The harness filled its candidate arm with cars that were never candidates and got zero nulls in 144 pages, contradicting two hand-verified soft 404s. A reconstruction that answers confidently in the right units is worse than none, so it was deleted rather than patched and the measurement moved inside the real check.

The state medians reach the model page, through one component

06:53 am AWST · 5b676b3

The ticker's used-ask range section gains a By state block: the same range the bar above draws, decomposed into each state's own deduped median against the national one, with n per row and the duty caveat riding along. It lives inside the range section rather than in a new grid cell, because it is that section's figure split eight ways, and a narrow list in its own full-width panel would be the empty-space complaint this page just got rid of.

Both surfaces now render through one StateMedians component, so the valuation page and the model page cannot drift apart in what a state The valuation keeps its subject-state highlight; the ticker, having no subject, highlights nothing. On year routes the block says all years, like the book beside it, because states are not year-faceted.

Where it's priced: per-state ask medians join the valuation

06:48 am AWST · b4d6010

The board's data object gains a states array (migration 0139): per-state used ask medians for every model, one price per physical car within each state, the eight real states only, rows printed at 8 or more cars. A car a dealer group advertises in two states sits on both states' shelves, because it is genuinely on both, while the national figure counts it once. Validated to the dollar against live_snapshot before anything read

The valuation result reads it as 'Where it's priced': each state's median with its delta against the national deduped median and its n, the subject's state highlighted, and a stated absence when the subject's state holds fewer than 8 cars. The caveat rides with the figures: asks mix drive-away and excluding-charges as advertised, and state duty schedules differ, so part of any gap is government charges rather than the car. On Ranger the spread is real and large: WA at $37,990 against NSW at $52,990 around a $42,990 national median.

The ticker payload carries the array for the model page to use when it wants it. Also refiles 0138 verbatim from the ledger for the active crawl session, per the interleave runbook.

The valuation answers the depreciation question, with what we hold

06:33 am AWST · 530cec0

The age ladder joins the valuation result: the subject model's used asks by year, today, drawn from the same board the model page's Vintage Ladder reads so the two surfaces cannot disagree, with the subject's year marked and each adjacent-year step stated with both ends' n. On a 2021 Ranger it reads: 14.6% off the newest used year, around 3% a year through the generation, 16.1% at the generation change. Every step is a cross-section of today and the copy holds that the forecast version appears when months of observed history have earned it.

The odometer-vs-price panel stops rendering a titled empty box: it draws the chart or states which of its two conditions failed (no odometer given, or under three comparables carrying both figures). The same discipline applies inside the new ladder, whose thin states explain themselves rather than leaving a rung frame around nothing.

Around the scatter, two figures the engine already knew but never dollars this comparable set sheds per 10,000 km, computed by the same robust fit as the model page and refused below the same 6-car, 20,000km bar. The range hero gains its market position: where the mid sits among every used ask on the model's board today.

Steps only print where both years hold 8 or more used cars, because a step between two thin medians is noise wearing a percentage. All four Ranger steps hand-verified against the board's own medians.

Docs: cost the client-rendered option instead of leaving it open

06:06 am AWST · 8682263

COVERAGE-CEILING listed "read the JSON those pages already fetch" as the first option against the rendering barrier. It is only worth doing if those sites share a platform, because then one reader unlocks many.

WordPress, carplace is Astro, and firstchoicefreo, coastlineautos and eliteautodeals advertise nothing. Per-site reverse engineering for about one dealer each, against unofficial endpoints that can change without notice.

canningvaletoyota would nearly justify it alone at 1,000+ cars, and its robots.txt explicitly welcomes the endpoint its own pages call. Recorded as the best candidate if this is ever picked up, and explicitly not picked up now: the same effort spent on sources that serve stock in HTML reaches more cars.

An option costed is more useful than an option left open.

Discovery: two more inventory paths, and the wall behind them

05:13 am AWST · 893ba2e

The marketplace channel's weak link was guessing a domain from a trading name, so the upgrade is to look the dealer up. It works: "First Choice Freo" resolves to firstchoicefreo.com.au with a /stock-list page holding ~124 vehicles.

The instructive part is that the earlier probe already had that domain right and still reported "no recognised inventory page" — because /stock-list was not among the paths it guesses. Domain right, path wrong, verdict blamed the site. That is the second time, after /vehicle-inventory on canningvaletoyota. Both paths are now in the list, and the lesson is written where the list is: it is a list of guesses, so a verdict derived from it is a statement about our guesses first.

The source is still unusable. Its stock page has no JSON-LD, no framework payload and no listing links — client-rendered, like canningvaletoyota, carplace, coastlineautos, morleycityhyundai and eliteautodeals before it.

So search fixes *finding* sites, which was a real limit and is now a smaller one, and the remaining barrier is rendering, which better discovery cannot touch. Recorded in COVERAGE-CEILING with the options unchanged.

Batch 4: polish, the subset that earns its diff

04:16 am AWST · 6184c78

The home tapes run the same length instead of one always ending two rows short; coverageStats joins the tolerant regime so a failed fetch costs the hero stat rather than throwing through the whole regeneration and serving ever-staler HTML; and the hero and coverage panels hide rather than render zeros when it is down.

The by-make table stops being 3,700px of long tail: 25 brands on screen, the remaining folded behind a native details element, no client JS. The two-price panel says when every row is one marque and one group's syndication, an anecdote rather than a pattern, derived from the rows rather than asserted.

/moving gains the one freshness stamp it never had while its siblings stamp every panel, and its per-model medians print whole days like the aggregate beside them instead of mixing precisions in one table.

Coverage rows carry their recorded disabled reason (the hero has promised 'a source that blocks automated access is recorded' all along), a merged target that shares the row's own name falls back to its id slug instead of reading as merged into itself, the count cells gain thousands separators, and the extracted-sources eyebrow reads as a count of sources rather than of cars.

A delisted listing's yellow hero figure says it is the final ask before it left the market, not a current price. /api, dealer and listing pages get their own titles instead of the default. A malformed API request no longer burns a quota unit for the mistake its 400 is reporting, with auth still checked first. Dealer stock-mix percentages round by largest remainder so the three shares always sum to 100.

Deliberately left alone: the sales tape's burstiness (that is what per-source confirmation batches honestly look like, and the yard names now distinguish rows), the API's mixed key casing (a breaking change for existing consumers, one for a versioned v2), and the home grid's latent imbalance when a conditional panel drops out (a rare state that wants visual iteration, not a blind fix).

Batch 3: the client-facing UX list

04:11 am AWST · a50c92d

The coverage table sorts by the name a reader scans for instead of raw source id (35 out-of-alphabetical breaks made 731 rows unscannable), gains a sticky header on desktop by unwrapping the horizontal scrollbox where the table fits without it, and stamps its timezone in the column header, because east-coast clients were reading Perth times three hours wrong.

Confidence gets three visual states for its three labels: high a hot solid band, medium a deep muted solid, low a wide outline. The old treatment went solid-hot at 0.5 while the label switched at 0.75, so a 0.73 medium was pixel-identical to high.

The valuation scatter gains the price axis it never had, three labelled ticks at d3's choosing, so a reader can finally put a dollar figure on a point. A rejected input no longer renders its refusal above the 'enter a make and model' empty state, contradicting itself.

Reports joins the shared nav (it was reachable only through one link on /market), the reports index drops to one h1, and the embedded week render loses its back-link to the page it is on.

Figures read as figures everywhere: the by-make and discount-band counts and the insight's n gain their thousands separators, the insight workings group their counts at the generator, and the report and dealer hero stats join the terminal's mono figure face instead of drifting to sans. The listing hero date joins the site's day-month format instead of 12/08/2026, and /support joins the shared heading and body scale.

Batch 2: copy that outlived its data, and claims made true again

04:08 am AWST · e101beb

Stale and false statements from the audit, each replaced by something the data supports or by the data itself.

Days to sell means tradeable stock now (migration 0137): both turn_by_model and the sale-speed board exclude new-car delistings, whose age counters reset with allocations rather than buyers. The /moving ranking that led with 'Skoda Kodiaq, 0 median days' built from fifteen cycling new cars leads with real used turns, and the /moving hero plus the home vital say the basis.

The valued-recently feed answers what it claims: it starts from the hygiene-filtered request log rather than stored rows, because seven of its ten most recent subjects were written by requests classified as bots, our own verification runs included, and a usage feed that is mostly the crawler watching itself is not a usage feed.

The API docs stop lying about themselves: the sample response derives model_version from the engine's own constant instead of pinning 0.6.0 against a live 0.8.0; the quota claim says which responses carry the field; and the door gets a handle, with keys requested via the same address the crawler already publishes in its user agent on every request. Unknown and malformed listing ids return a real 404 like the dealer pages, instead of a friendly paragraph with HTTP 200 that taught caches an invented id exists.

Coverage stops asserting an ordering the data inverted (722 here vs 698 on the home page read as a broken promise), stops crowning CarsGuide the one crawled marketplace now that it is not, explains the Rate column whose 12 seemingly impossible rows were backlog rather than arithmetic errors, and marks counters that come from an older run than the shown crawl time, which the RPC has reported all along.

/moving's aged columns name their denominator (age-dated used, not all live used, which overstated Karoq's stuck share as 58 percent), the colour panel opens with its real population instead of an all-conditions count over a used-only analysis, its platform claim matches the two platforms that actually publish an age, and the thin- filter copy stops promising 'the sales below' above an empty table.

The comparables table heads with 'the 30 highest-weighted of N' so its header and its rows agree, relabels DOM to Watched (d) for what it measures, and marks drive-away asks as restated so the raw label never wears a converted number.

Batch 1: every number agrees with its neighbours

03:47 am AWST · badedae

The audit's misleading list clustered around one failure: the same statistic wearing two values, two units, or two definitions in one viewport. This batch ends the disagreements.

One definition of sold, everywhere. market_pulse now counts cars the way the weekly report does: one per physical vehicle at its final exit, no listing of it still live, value as its final ask once (migration 0135). The header, the home hero and the reports page can now differ only by their stated window, never by their unit. Repricings likewise count once per car per UTC day. The headline moved from $181M of listing exits to $91.5M of cars, which is smaller and true.

One read of the pulse. The chrome and the hero memoised separate reads instants apart and printed $181.2M beside $181.0M for one statistic; marketPulseShared() puts both behind one cache key, and the duplicated moneyCompact formatter that dressed the two reads collapses into lib/format. Verified: both occurrences render byte-identical.

Windows say their names: the chrome reads 'confirmed sold, last 7 days' rather than 'this week', because the weekly report's calendar-week figures share a screen with it. The hero's big count is labelled 'listings monitored', because listings are not cars and the site says so itself two panels later.

Denominators own their remainders: the condition mix names the share with no stated condition instead of summing to 92 and stopping; the segment table states it covers 168,717 of 177,254 live listings and why; the comparables line reads '430 used (414 live, 16 delisted), of 721 found' instead of bracketing numbers that did not sum to their own headline.

Computed, not asserted: the cuts-vs-rises caption derives its direction and no longer dangles when rises are zero or lies when rises win; the insight headline states one decimal so it can never disagree with its own workings by rounding; the stale hardcoded price-basis fractions are gone; the valuations vital says what it counts (requests answered, honest refusals included).

Twins get names: identical tape rows for two same-model cars sold in one crawl batch now carry the yard that sold them, truncating rather than widening so phones stay at 375px. The states panel loses its per-minute Freshness chip, which claimed seconds over a six-hourly board; the page-top dateline is its honest clock.

Also files 0136_pass_ordering_inputs verbatim from the ledger for the active crawl session, per the interleave runbook, keeping the drift check green.

Ops: measured the pass-ordering question, and the answer is leave it alone

03:43 am AWST · ea9ab20

OPERATIONS recorded dispatching long sources first as the proper fix for the pass tail, deferred because the current order buys something makespan does not price: interruptions must not starve the least-covered sources. That is not a hypothetical — a fixed order once left 88 of 183 sources never visited.

eval-pass-ordering.ts simulates the worker pool on real per-source durations, each source's own measured seconds-per-page times the pages its budget allows. 700 sources, 281.4 worker-hours, 24 workers, perfect packing 11.73h:

stalest first (ships) 12.71h cut at 4h: 521 never started longest first 11.73h cut at 4h: 653 never started

Longest-first hits perfect packing and buys 8% — one hour — for 132 more unvisited sources on an interrupted pass. Not worth it.

The interesting part is *why* it is only 8%. The four-hour cap already took the win. While one source could run 24 hours it dominated the makespan and reordering would have mattered a great deal; with none able to exceed four hours the shipped order is already near optimal. Fixing the budgets removed the reason to touch the ordering — which is the better outcome, because the ordering is load-bearing for fairness and the budgets were not.

Third measured negative today, after trusting `found` on parse rate and stripping descriptors from badges. All three looked obviously right.

Ops: listing_observation is the next table to watch, and why it is different

03:07 am AWST · 7a4c9de

With raw bodies gone it is the largest thing in the database — 2,688 MB of 3.9 GB, 2,015 MB of that the `fields` payload.

The dedup is already working: 680,606 of 942,494 observations store an empty payload because an unchanged field set records `{}` and the previous sha. Only the 28% that changed carry content.

Recorded mainly to head off the wrong conclusion. This is not raw bodies again and must not get the same treatment: `fields` is on the read path, recovered by listing_current through a lateral whenever the newest observation is empty, so moving it to object storage would trade a storage bill for latency on the thing users wait for.

The number to watch is the non-empty share, currently inflated by growth — 50,000 listings arrived in the days before this measurement and every new listing writes a payload. If it does not fall in steady state, the questions are whether `fields` carries something that belongs in a column and whether the parser emits spurious differences that defeat the sha.

No action. Written down so this is read as a trajectory rather than discovered as a surprise, which is how the 8.5 GB of raw bodies was found.

Valuation: the request counter stops blocking the dedupe read

01:37 am AWST · 510d2c8

After the answer is computed there are three sequential round trips of bookkeeping the caller waits on: log the request, check for a recent identical valuation, store this one. The counter informs none of it — it swallows its own errors and returns nothing anybody reads — so it had no reason to go first.

`await logRequest(...)` then `await select` becomes `await Promise.all([select, logRequest(...)])`. Two hops to Sydney become one.

**Not measured, and I am not claiming a number.** A full crawl pass started at 00:43, and timings taken under it (1,564ms for the Ranger) are worse than the idle baseline this change was built against (1,061ms) — which says load, not regression. I made exactly that mistake earlier today, concluding latency tracked comparable count from numbers taken under crawl, and had to correct it. Once is enough.

What is verified: estimates unchanged, days-to-turn unchanged (51, 34, 27, 29), records still stored and deduped. The change removes a sequential await and can only be slower by noise. The number can be taken in the next quiet window.

Valuation: days to turn no longer waits for the comparables

12:37 am AWST · ce5283b

On an idle database a valuation costs about a second whatever the cohort size — 434 comparables 1,173ms, 97 comparables 1,003ms. That floor is sequential round trips to Sydney at roughly 250ms each, not payload and not memory, which is the real reason more RAM would not have moved it.

deriveDaysToTurn depends only on the subject. It was awaited after every comparable had been fetched, weighted and adjusted, for no reason beyond where the line sat. Started alongside the comparable fetch and awaited where it is used:

Ranger 2022 1,173ms -> 1,061ms Hilux 2021 1,052-> 936 RAV4 2022 980-> 803 CX-5 2021 1,003-> 767

Roughly 15% off, up to 24% on the smaller cohorts where the fixed cost dominates most. Estimates unchanged; days-to-turn values unchanged.

This also corrects something I told Taj earlier. Measuring under crawl load, latency appeared to track comparable count and I said so. On a quiet database it plainly does not — CX-5 with 97 comparables took as long as RAV4 with 186. The load was the variable, not the payload.