Every change, in the words it was made with

Make the probe check what its own header claims it checks

11:52 pm AWST · 72772ba

Seeding decisions were made on link count. That is not enough, and two sites seeded tonight prove it: carlinandgazzardldv and carlinandgazzardvolkswagen each offered a dozen links that looked like stock, and every one was a model brochure — /new-vehicles/ldv-t60-max-ute/, /new-vehicles/id-4/. They have no used inventory on the web at all, and now sit at 0% on /coverage.

mazda.com.au was the same shape and only avoided seeding because it was checked by hand. The rule should not be that someone happened to look.

So the probe now opens the first discovered URL and requires a real car with a price on it before reporting a host as usable — which is what sweep 3's own header already claimed was done. Both discovery paths get the same treatment, the guessed inventory paths and the homepage-link fallback.

A brochure page no longer ends the probe for that host either: /cars can be model pages while /used-cars holds real stock, so it records the finding and keeps trying the remaining paths. "Links lead to model pages" is reported distinctly from "no inventory page found" — one is a dealer with nothing online, the other is a bad guess at a URL, and they want different follow-up.

Enter the marketplaces at 92 and 55 points instead of 52 and 8

11:48 pm AWST · db8d368

Both sites cap a result set however deep you paginate, so coverage comes from entering the catalogue at many points rather than paging one further. CarsGuide goes from 52 facets to 92 and Autotrader from 8 to 55, all makes.

Every facet was fetched and checked before being added, because a facet that ignores its path is worse than none: it spends the fetches and returns cars we already hold, while looking like expansion. Two shapes failed and are deliberately absent. CarsGuide /buy-a-car/{state}/{make} answers 200 and does not filter — /qld/ford returned Toyota, HSV, BYD, Audi, Honda, Mercedes-Benz and nine other makes, so the obvious 8x40 grid would have been 320 facets of duplicates. Autotrader /for-sale/{state}/{make} 404s, and its /for-sale/under-10000 is disallowed in robots.txt and was left alone.

What passed: every make facet returns only that make, checked against the make segment of each listing URL it offered. CarsGuide's existing location facets are real too — sydney-location returned 26 cars, all NSW.

The script records a note that cost some time to learn: probe these with extractorFor(source.platform), not with whichever extractor matches the HTML. Both sites embed JSON-LD, so the auto-match hands back the generic jsonld extractor and a different, misleading set of URLs — which is how /qld/ford first appeared to contain filter links rather than mixed makes.

Committed rather than left in the database, because config that exists only there is config the repo cannot rebuild.

The picker's last route to a badge that year never had

11:30 pm AWST · 6afe0bd

Taj reported the valuation picker offering badge/year combinations that do not exist. Migration 0040 fixed the main path — badges narrow by year and years by badge, so the two constrain each other whichever order they are chosen in. This is the hole it left.

When the narrow list comes back empty the picker falls back to the model-wide list, so a rare combination never leaves the user with nothing to choose. For badge-within-year that hands back the exact problem: 168 of 3,613 model-year combinations in the corpus have no badged example, and for those the fallback offers every badge the model ever wore. The 2019 Wildtrak again, by another route. 332 live listings sit in those combinations.

Badge is optional, so offering none is a real answer and the select already disables itself on an empty list. Offering one we cannot value is not an to choose a badge and then told there is not enough data for it.

Left the fallback in place for transmission, drivetrain and fuel, where an empty list is a dead end rather than a fair "we do not know".

Four more defects for the quality log, and what they share

11:27 pm AWST · bb54393

Entries 9 to 12: the pipeline that died when the corpus outgrew OFFSET, the rotation that re-fetched the same two hundred cars, the extractor that never handled the type its own header claimed, and the dead URL that ended a whole traversal.

The first eight entries are wrong numbers. These four are absences that looked like completeness — a read that returned less than it should and said nothing. Three are the same root cause in different clothes: a result set truncated by a cap or a timeout, with the error dropped or never checked.

Worth its own line, because it is the uncomfortable one: entry 9 broke with no code change at all. It worked at 20,000 listings and failed at 50,000. A control that degrades with scale gives no warning, and the day it breaks is not the day it changed.

One dead facet no longer costs every facet behind it

11:26 pm AWST · c254856

A non-200 index page broke the whole traversal. Autotrader's for-sale/nsw/page-2 returned 404 and the seven remaining state facets were never fetched — the run recorded 228 listings and stopped. CarsGuide queues 52 facets behind the same loop, so its first dead URL ended the pass there too; both marketplaces have been sitting at ~226 listings each.

Skips the dead URL and carries on now. `truncated` still gets set, so the pass is still recorded as partial and still refuses to infer any delisting from it — continuing changes how much we read, never what we are willing to conclude from it.

Found while probing whether Autotrader supports facets beyond the eight states it has: /for-sale/used/{make} works, /for-sale/{state}/{make} does not, and /for-sale/under-10000 is disallowed in robots.txt and was left alone.

Stop six sources spending every pass on brochure pages

11:20 pm AWST · 234a456

Six sources showed 0% coverage and 100% failure — Alice Springs Toyota, Watson Toyota, Darwin GWM, Riverina Volkswagen, Dubbo Ford, Autosports Group. Nothing was broken about them. Discovery was queueing /new-vehicles/haval-h6/, /new-vehicles/id-4/, /vehicles/f-150 and /new-vehicles/suvs-4wds/ — model brochure and category pages — and the extractor was correctly returning null, because there is no vehicle on them to extract. Returning anything would have invented a listing, which is what excluded mazda.com.au from seeding earlier tonight.

The detail matcher asked for one digit after the category segment, and every one of those model names has one. A real listing carries a stock number or a model year and so has a run of four or more; a model name's digits run one to three. Four now.

Measured before changing it rather than after: all 957 listings currently held from jsonld sources have a four-digit run, so this drops none of them.

The other eight sources in that group are already fixed by the Product change and were re-crawled to confirm it in production — Trivett and Castle Hill Renault at 100%, Kerry Nissan and Riverina Nissan at 92%, from zero.

The miner could not see .au, .net.au or .com dealers

11:14 pm AWST · 722f493

It matched .com.au only. That is most of the Australian market and not all of opened to registration in 2022, The Yard WA on a bare .com. Five of our own 263 sources are hosts this miner could never have proposed — it has been the best discovery method available while being structurally blind to them.

Longest suffix first in the alternation, so bartons.net.au is captured whole rather than truncated at .au. The subdomain-depth guard now counts against the registrable domain instead of a flat three labels, so a two-label suffix is judged the same way as a one-label one; the platform CDN hosts it exists to reject still are.

Also adds --top. The first deep run found 161 hosts above the threshold and printed 60. The hundred it dropped were never probed and nothing on screen said they existed; it now says so and the limit can be raised.

DEPTH-PLAN: mark the build-now queue shipped

10:58 pm AWST · dafaf2e

Does colour cost you time? The controlled answer: barely

10:56 pm AWST · 6dd616b

colour_board joins the 6-hourly refresh (0063): each used car's seller-published age residualised against its own model's median, so a colour that skews European or dear cannot masquerade as slow. The result is the honest headline — residualised, nearly every colour sits within a day of its model's typical age (white 0, grey 0, black -0.6, red -0.3 over thousands of cars); the raw 7-day black-vs-red spread was mix. The panel on /moving says a small spread is a real an ordered rule list, black before pearl; live-listing ages, not completed sale times, and the surface says both.

The rotation that kept re-fetching the same 200 cars

10:56 pm AWST · 664437e

Taj reported Westside stuck at 200 extracted against nearly 2,000 on their site. Two separate things were behind it and only one was the budget.

The detail budget rotates unseen listings to the front of the queue and then the stalest, so a capped pass makes progress through a yard instead of re-reading the same cars. It builds that order from a query of every listing the source has — unpaged, and with its error dropped. PostgREST returns 1,000 rows and no complaint, so at any yard holding more than a thousand cars every listing past the cap was absent from lastSeen, counted as never-seen, and sorted to the front on every pass. The same few hundred were re-fetched each time and the rest of the yard was never reached.

Westside holds 1,229, Bartons 1,203, Country Cars 1,203, Brisbane Cars 1,164 — every yard big enough to need the rotation was big enough to break it.

Paged by key now, error read.

Worth being straight about what this does not fix. Those sites declare a Crawl-delay and a full pass measures ~10 seconds a page, so Brisbane Cars' 6,485 discovered listings is about eighteen hours of polite fetching however the queue is ordered. Ordering decides whether a capped pass makes progress or stands still; it does not raise the ceiling. The ceiling is a throughput question and it is Taj's call, not a code change — the crawl-hosting decision already open.

Read the Product-shaped vehicle pages this file claimed to read

10:53 pm AWST · 086f90b

Eight sources discovered listings and extracted none of them — found 15, failed 15, a permanent zero on /coverage. Castle Hill Renault, Kerry Nissan, Buckby Motors, Llewellyn Subaru, Darwin GWM, Alice Springs Toyota, Dubbo Ford and Autosports Group.

The cause was a gap between this file's header and its code. The header says it handles Vehicle/Car/Product; isVehicleNode never included Product. Several platforms — i-Motor among them — publish vehicle pages as a bare Product carrying only name, brand, image and an offer, and every one of those pages parsed to null.

Product alone is not evidence of a car: it is the type an accessories shop uses for floor mats, and carmatewarehouse.com.au, seeded tonight, sells cars under a name that suggests otherwise. So a Product is accepted only on evidence — a field only a vehicle has, or a brand the taxonomy recognises together with a plausible model year in the name. Verified both ways: rubber car mats branded "Car Mate" and roof racks named "2022 Model" are rejected; a 2025 Renault Duster and a Corolla whose marque appears only in its title are kept. Castle Hill Renault now yields make, model, year, badge and $34,990 where it yielded null.

Titles are parsed only as a fallback, never over a field the site published, and the remainder after the model stays whole in the badge — series and drivetrain are guessable from "X1311 MY25 4x2" but guessing them would invent precision the title does not have.

The same work fixed a badge defect found on the way: sites that publish the whole title in vehicleConfiguration were sending "2022 Nissan X-TRAIL Ti-L e-POWER T33" through as a badge, which reaches the valuation picker as an option no second listing can share. Now "Ti-L e-POWER T33".

Also records eight national locators and auction houses as surveyed and left alone. The hope was that one Toyota or Lexus locator would cover a whole network at once; none of the 18 probed is available to an honest crawler — four disallow the path, two answer 403 or a challenge, the rest publish no inventory. Dealer-level sites remain the way in.

Same car two prices, and what a demo saves — badge-matched

10:51 pm AWST · 1ae6635

vin_two_prices (0062): one VIN live at different asks, top spreads. The naive board's biggest rows were dealer DMS errors (a wrong VIN sharing an MU-X and a GMC Yukon), and VIN-merged vehicles make a make/model guard impossible by construction — so the guard is lower ask, and both listings are linked so every row is checkable. Top verified row: an Alpina XD3 at identical kilometres, $16,000 apart across one group's sites.

demo_gap: demo-vs-new advertised gaps, badge-matched with both sides n>=5 — the model-level raw gap inverts on models whose demos skew to high trims (D-max: raw negative, five of six badge cells positive). Cells where the demo is a model-year older say so on the row. Both panels on /market with their guards written on them.

Page by key, not offset — the corpus outgrew OFFSET

10:45 pm AWST · da3f153

compute-insights has been failing outright since the corpus passed about 21,000 priced listings: "canceling statement due to statement timeout". The market insights on screen were the last ones that succeeded.

The cause is not the query but the paging. OFFSET makes the database produce and discard every row before the window, so page fifty costs fifty times page one — and listing_current resolves deduped fields per row, so that cost is paid again on every row thrown away. It worked at 20,000 listings and stopped working at 50,000, which is the kind of failure that arrives without a code change.

pageByKey asks for "the next 1,000 rows after this id" instead. Constant cost per page, because the index seeks straight to the key. compute-insights now reads all 47,335 priced listings in 45 seconds, where before it read none.

All four offset-paged readers are converted: insights, the delisting check in store.ts, and enrich's whole-table loads, which page a vehicle table already past 42,000 rows.

Two further things this surfaced. insights capped its corpus at 50,000 rows, which the corpus was about to cross — it is 200,000 now, and it is a real cap that throws rather than a silent truncation. And pageByKey never swallows an how a source count of 287 was reported against a true 751 earlier tonight.

Also seeds 29 dealer sites from the deep link mine. mazda.com.au probed clean and is deliberately excluded: its 14 "listings" are model brochure pages with no price, odometer or stock number, and seeding it would have invented listings that do not exist.

Ticker v3: the honest float, Today's Range, time-and-sales

10:44 pm AWST · 0cfb30d

The hero now leads with cars, not listings — 2,165 Rangers behind 2,823 listings, because dealer groups syndicate — and Today's Range draws the five-number summary of used asks at one price per physical car (0061; a listing-weighted median ran ~6% hot). The repricings list becomes time-and-sales: cuts, rises, confirmed exits and relists, with first-seen events deliberately excluded — they are mostly our own coverage growing, and showing them would fabricate a surge. The pre-registered photo/description-change Tell is announced on the panel and publishes, whichever way it falls, at 2,000 observed cuts. Ticker fetch memoised (was ~6s cold on the biggest models); live reads on the lean view.

Ask each source's sitemap how big its yard really is

10:38 pm AWST · d14f89e

Discovery walks a site's pagination, which stops wherever that site decides to stop. A sitemap is the site's own list of every page it wants indexed, costs one fetch, and has no pagination to run out of — so it is both a truer inventory count and a cheaper one. crawl.ts already reads sitemaps; 143 of 236 sources are pointed at one and 93 are not, and nothing until now measured which of those 93 were missing out.

John Hughes holds 751 listings here against 3,136 URLs in its sitemap; Westside 1,229 against 1,808. Duttons at 359 against 351 is already complete, which is the answer worth having too.

--apply writes the sitemap in following the convention the other 143 use: indexUrl is the sitemap, indexUrls holds the HTML pages that yield listings inline, because crawl.ts queues those first so a truncated pass still refreshes prices instead of only discovering links.

Counting what we hold went wrong twice on the way and both are worth naming. Paging listing_current by .range() without an order returned overlapping pages and undercounted John Hughes as 191. Ordering it fixed the ordering and exposed the real fault: a statement timeout at offset 21,000, because the view resolves deduped fields and a deep OFFSET redoes that work for every row it discards. The loop dropped the error and broke on the null, reporting 287 against a true 751 with nothing to say it was partial. Counted per source now, one head request each, every error read.

The lean view: aggregates stop paying for fields they never read

10:36 pm AWST · 6a9c4bc

The fields-erosion fix made every listing_current reader pay the sha back-lookup lateral whether it touched fields or not — a LATERAL with LIMIT cannot be join-removed — and market_by_make went from seconds to 30s, 500ing home and /market on a cold cache. listing_current_lean (0060) drops the fields resolution and the per-row event count and has no fields column at all, so nobody reaches for it and silently reads eroded state. Seven cross-sectional aggregates move onto it (market_by_make 30.4s -> 1.6s measured); fields readers stay on the honest view, where the delisted set is small and pg_cron has no gateway. The interim model_watch functions are dropped — the boards made them redundant before they were ever called.

Four dormant signals surface: watched, aged, marked down, on-roads

10:30 pm AWST · e10f3b2

The fields-erosion fix made listing_current honest and slow — a full live-scan jsonb aggregate now costs ~57s — so the fields-dependent surfaces read precomputed boards, refreshed by one in-database pass (refresh_depth_boards, pg_cron 6-hourly with the crawl; 0059). One scan feeds all four:

- Most watched (home + ticker hero): the dealer site's own page-view counter over the seller's own listing age, used stock, n>=10 cells, days_old>=7 so a first-weekend spike cannot mint a rate. 34,685 listings carry the counter; platform basis stated on the panel. - What is not moving (/moving): used stock 180+ days by the seller's counter — 5,801 aged listings, condition-split on the surface because 76% of the aged cohort is new/demo allocation, not resale. - Marked down hardest (home): dealer-stated was-prices, labelled as the marketing claims they are, cross-referenced to our observed cuts one panel up. 8,152 live listings show one; median cut $3,000. - What on-roads cost (/market): drive-away vs EGC measured on 8,380 same-document pairs, both sides >=$1,000 (zero-valued placeholders would fabricate the gap), per state at 300+ pairs from 5+ dealers.

Count valuations served, and keep the counters on every screen

10:17 pm AWST · 61cbf37

Taj tracks usage by the header counters, so the home-route suppression is reverted — they stay in the chrome everywhere. And the count now means what he needs it to mean: valuation_request (0055) logs every answered request append-only — web and API, ok and honest refusal alike — where the valuation table deduped repeat subjects and never stored refusals at all. Backfilled from stored valuations as a provable floor. Clicking Value moves the number.

Paginate each marketplace the way its own robots.txt allows

10:11 pm AWST · 79c4739

Autotrader's first real pass got 228 listings and a 404 on every second page. The clean /page-N path that CarsGuide serves does not exist there, and it was assumed rather than checked — the check had been done on one site and the result applied to both.

They differ for a reason that is in robots.txt, not in the markup. CarsGuide disallows every query string, so ?page cannot be followed and /page-N is the way in. Autotrader disallows only named parameters — store, sortBy, dealerId, distance, keywords — so ?page is permitted and is the only thing that works. Each site now gets the form it actually allows, verified three pages deep on 78 and still going on Autotrader.

Also fixes the miner reading the same thousand captures every time. --limit 8000 returned exactly 1000 rows because PostgREST caps a response there, so "mine deeper" mined precisely as deep as before and reported the same hosts. Paged now. That cap has quietly truncated four different things in this codebase; it is worth assuming it is present in any query that could return more than a thousand rows.

Twenty new sources seeded from the first two mining passes. 234 dealer sources, up from 181.

Daily rollups begin tonight, in-database

10:11 pm AWST · e59de4a

daily_model_rollup (467 models at the live>=5 floor, used-stock quantiles only — blending conditions is how '0 yr, 12 km' happened) and daily_market_rollup, written idempotently by write_daily_rollup() and scheduled with pg_cron at 15:55 UTC daily — no Actions minutes, no laptop. Day one is already in the books. This table is the precondition for every deferred trend surface in DEPTH-PLAN: the 4-week range labels, weekly exit rate, views flow, and the mix-adjusted index itself.

Depth batch 1-2: one definition of advertised age, person-fields scrubbed, one job per page

10:06 pm AWST · 76f61f1

advertised_days() (0052) is now the single reading of how long a car was advertised: it prefers the seller's day counter and falls back to imotor's createdAt aged to the delisting — verified sane first (zero future dates, median 54.6d against Dealer Studio's 51) — so ~4,000 imotor listings join every turn view instead of falling out for want of a mapping. Six SQL readers and the TS advertisedDays() now share it; sixteen test cases cover both paths.

Person-fields are refused at the ingest choke point and scrubbed from the 69,701 stored observations that carried Dealer Studio's buyer_name key (one arrived populated — somebody's name from a dealer's DMS does not belong in a corpus about cars; 0053).

The 'too much' cuts from DEPTH-PLAN: /market drops its echo panels (tape, turn table, sale-speed trio live on the terminal and /moving) and keeps what is uniquely its own; home loses the individual-rises fallback; the credibility bar hides on the home route where the hero already states it, and carries the confirmed-sold definition as a tooltip everywhere else. Price-basis disclosure added to ticker heroes and the market medians.

The depth plan: what the corpus holds that no surface reads

09:49 pm AWST · 4f07bbc

From a 21-agent research pass — live-DB inventory, surface audit, prior art, moat audit, four ideation lenses, editorial merge, and one adversarial verifier per candidate. Ten build-now features (each with its verified counts and its mandatory guards), four time-gated ones with SQL-checkable unlocks, two killed or reduced by verification (per-car overpriced rankings are a badge/year ranking in a mispricing costume; the raw Spread dissolves under age+km control), and the cuts that answer 'is anything too much'.

Find dealers in the pages already fetched — 94% of them work

09:48 pm AWST · c1ec2f7

Every previous way of finding sources was a guess. Inventing domains from group names ran at 35%, then twice at zero. Deriving them from dealer names in the marketplace directory ran at 25%. Reading the AdTorque client list ran at 60% and was exhausted in one sitting.

Dealer sites link to their group's other yards. Those links are sitting in raw captures already on disk, and reading them costs no fetches at all. scripts/mine-linked-dealers.ts pulls hostnames out of stored HTML, drops the platforms and infrastructure and anything already seeded, and ranks what is left by how many of our sources link to it.

The first pass over 1,000 captures found the Bartons group's fourteen franchise sites, Llewellyn's three, Kerry's two, and the Riverina group's four. Probed: 15 of 16 usable, and 6 of 10 from an earlier batch. 21 new sources tonight from data we already had.

Two rules that make the output worth probing: a host reachable from more than one of our sources ranks above one mentioned many times by a single site, because the second is usually a footer; and anything with more than three labels is a platform subdomain rather than a dealer.

229 dealer sources, up from 181 when this started.

Paginate the marketplaces without touching a query string

09:21 pm AWST · 103e2c3

A facet page yields 26 cars and NSW alone lists 23,240, so the marketplaces were being read one screen deep and stopping. Both paginate with ?page=N, and both disallow query strings in robots.txt — following the links they render would be crawling paths we were asked not to.

The same pages are served at /page-N, which is permitted and verifiably extractor offers that instead, and only when a page actually filled, since a short page is the last one.

Which exposed the next problem. Tasmania holds about 52 cars and serves page 3 as a repeat rather than a 404, so a "next" link exists forever and the crawler would page until its budget ran out, re-reading cars it already had. Pagination now stops when a page contributes no new listing — a general rule in the crawler rather than a special case in one extractor, because a page that adds nothing is the end of the facet whatever the site claims.

Index budgets to 400 across 52 CarsGuide facets and 8 Autotrader ones, detail budgets at 2,500 each. Detail fetches are the real cost at 4s apiece, so that is the honest ceiling per pass: about 2,500 new cars per marketplace, with unseen-first rotation carrying the rest to the next one.

Autotrader is reachable after all — seeded, on the extractor we already had

09:12 pm AWST · 629f9a0

The survey of 2026-08-10 recorded Autotrader as a Peakhour 403 on everything including robots.txt, and it was left alone on that basis. Re-probed today it answers 200: robots.txt is 3,296 bytes of ordinary rules and /for-sale/used returns real listings. The record is corrected in place rather than quietly replaced, because a source moving from blocked to reachable is worth knowing.

Its Disallow rules cover query strings — store, sortBy, dealerId, distance, keywords — the fuel-type filters, and paths of nine or more segments. Listing URLs are seven segments and permitted.

It shares CarsGuide's platform, down to the URL shape /car/{id}/{make}/{model}/{state}/{suburb}/{body} and an adjacent id space, so the carsguide extractor reads it without modification. One of the country's largest marketplaces, added for the cost of a probe.

Two bugs it exposed, both the same shape as the odometer one:

HTML entities were never decoded, so a dealer arrived as "K.S. & Sons Motor Company". Now decoded once, in the text pass everything else reads from.

Body type came back as "s Convertibles Dual cab utes Hatchbacks People movers Statio" — Autotrader lists every body type in its navigation, the spec parser matched the label there, and a menu was stored as the car's shape. Short spec values are now validated as short: one or two words, letters only. Anything longer is something else sitting next to the label, and nothing is the better answer.

Also recorded: easyauto123 and Suttons answer 404 while returning 180KB–740KB of markup — soft 404s from client-rendered apps. Not blocked, simply not served in HTML, which needs their JSON API identified rather than a better guess at a path.

Delete the map entirely, at Taj's direction

08:54 pm AWST · 2b788cf

Route, client component, projection and yard libs, both ABS assets and their ingest scripts, and the two database functions that served only them (0050 — the read path went first, so the drop breaks nothing). Dealer profiles survive as venue analytics and are now reached from the coverage table instead of the map. 0049 (committed here, authored by the concurrent alias session) is the model-spelling merge the ticker smoke test surfaced; migration files renumbered from the database's order of record.

Three more sources from the directory, and stop the probe trusting a form

08:53 pm AWST · c42e1db

Buckby Motors (TAS), Carz Direct (WA) and Alice Springs Toyota (NT), all found by turning names from the CarsGuide dealer directory into domains. NT is the thinnest state in the corpus at 497 listings, so that last one is worth more than its size suggests.

The probe was accepting an enquiry form as an inventory page: /enquiries/used-vehicle-enquiry/ matched on the word "used" and its navigation passed the link-count threshold. Pages whose path says enquiry, contact, finance, service, valuation, trade or book are excluded now, and Alice Springs resolves to a real listing page instead.

Honest yield: names from the directory converted to working domains at 25%, which is no better than guessing was. The directory's value is not that a name implies a domain — it is knowing which dealers exist and where, so the search is over real businesses rather than invented ones. Converting a name to a site reliably needs a search per dealer, which is the next thing to try rather than more derivation.

Note in VISUAL-PLAN that TERMINAL-PLAN supersedes the surface priorities

08:46 pm AWST · 237ba65

Build the dealer directory, and fix three things it exposed

08:43 pm AWST · 94fb9d8

128 distinct dealers now carry a name, suburb, postcode and state, read off the compliance line and applied to 202 listings without re-crawling a page — the raw captures were re-read instead, which is what retaining them was for.

The directory immediately earned its place. The ACT had **zero** listings in the corpus from 205 dealer websites and has 26 from here: a whole territory that was missing, found because the marketplace reaches yards whose own sites we had never located. Hobart, Alice Springs, Fyshwick, Derwent Park — the places a corpus of franchised metro dealers does not reach.

Three defects surfaced on the way, each caught by the data refusing to make sense rather than by a test:

An odometer of 1,116,981.5 km, to half a kilometre. The seller's description opened "for its age with only 111,698 kms. Fun and responsive 1.5L petrol", the spec parser matched the word Kilometres inside the prose, and stripping non-digits glued the two numbers together. A numeric spec now has to look like a number before it is treated as one; prose containing a number is not one, and nothing is the right answer for prose.

A car in the state of SUV. The URL fallback validated the state with /^[A-Z]{2,3}$/, which "SUV" passes, on a URL whose segments did not line up. Checked against the eight actual states now.

And the reason none of this appeared at first: dealer_name, suburb and postcode were written when a listing was first inserted and never revisited, so a listing first seen by a weaker parser kept its gaps permanently. Blanks are now filled on re-observation — blanks only, because a value already recorded is evidence from an earlier reading and replacing it would make the listing row quietly mutable, which is the opposite of how the observation log works.

Ticker pages: one market page per model, the scatter behind the number

08:39 pm AWST · 8fafd74

/model/[make]/[model] is the stock-ticker analogue: float and mix, used-stock medians, price against odometer with the valuation's own Theil-Sen fit (drawn only past its own guards — six cars, 20,000 km of spread — and clamped to the plot box), a by-year depreciation table, the model's repricing events, and its confirmed sales. Year chips filter the cloud per cohort via searchParams; thin cohorts print their count and refuse the rest.

0046 assembled it in one round trip; 0047 fixes what the smoke test caught before any page shipped: 'CX 5' and 'CX-5' share a slug, and matching one exact spelling served 1 car of a 593-car cohort. The ident set now spans every spelling, displayed under the one with the most live stock. Movers, most-traded and the market turn table now link through to tickers.

Home becomes the market terminal; the map leaves the nav

08:29 pm AWST · 13c966b

Taj's redirect: the stock market, but for cars. The home page now leads with the numbers the corpus can defend today — confirmed sold this week in dollars and cars, the 24-hour pulse, cuts against rises, movers boards both directions (Ranger cut 25 times at a median -$1,000; rises are rare and the count is the finding), the tape of individual sales and cuts, and most-traded models. A credibility bar in the header carries valuations served and confirmed-sold value.

What is refused, on the page, in words: a price index (needs weeks of history and mix adjustment — defect 5 was a raw average reading fleet composition as a price gap) and private-sale coverage (the corpus is 100% dealer sellers; the marketplaces carrying private stock block automated access). docs/TERMINAL-PLAN.md records the full mapping from stock-market idiom to what the data honestly supports.

The map route stays parked but unlinked pending Taj's call on deleting it; dealer profiles remain linked from it and by URL.

Seed three more groups, and read the dealer off the compliance line

08:26 pm AWST · c155198

Von Bibra, Bayford and Trivett, all multi-franchise groups on schema.org markup. Trivett is Inchcape's retail arm and is a second legitimate route to Audi and Volkswagen stock whose single-brand sites answer HTTP 400 to an ordinary request.

The bigger change is how the next sources get found. Guessing domains ran at 35% and then at zero across two batches; the AdTorque Edge client list ran at 60% and is now exhausted. So CarsGuide listings are parsed for the compliance line every Australian dealer ad must carry:

Licence no: MD27291, Next Ride Malaga, (08) 6150 3393, 26 Stanford Way, Malaga, 6090 WA.

That gives each listing a real dealer name, suburb and postcode instead of a state and suburb inferred from the URL — and the accumulated names become a directory of dealers that actually exist, which is a far better place to start than a list of domains someone guessed.

Six cases cover the parse, including that a page without the line returns nothing rather than a guess, and that the suburb is the suburb rather than the phone number sitting next to it.

Dealer profiles: one page per yard, used-stock medians headline

08:19 pm AWST · f38365a

/dealer/[slug]: inventory, make mix with per-make median ask, the sales we watched leave the market as a dot strip (median only past four aged sales, dots at any n), and locations. Headline medians cover used stock alone once a dealer holds five used cars — the first render blended Gardner Autos' 60%-new stock into '0 yr, 12 km', true of the listings and useless as a headline (0044). All data assembled by dealer_profile() in one round trip (0043).

The map: 448 dealer yards on an SVG Australia, no library, no tiles

08:08 pm AWST · 09a2fef

/map plots every yard — (source, postcode), because a dealer group's stock genuinely sits on different lots — sized by live inventory and coloured by median ask or days-to-sell on a validated single-hue ramp of the brand yellow. Turn mode draws yards below four aged sales as outlines rather than colouring a guess, the tooltip and a full table carry every figure colour implies, and the header counts what is NOT without one are left off rather than geocoded from suburb names.

Lambert conformal conic (18°S/36°S), coastline from the ABS STE layer regenerated at 1 km tolerance after the 5 km cut tore into shards at metro zoom. Aggregates live in Postgres (0042: map_yards, turn_by_source); the page serves them through the existing TTL cache.

Probe: ask the site where its inventory is instead of guessing harder

08:08 pm AWST · e9fb5c3

Eleven guessed paths still 404'd on three reachable dealer sites. The probe now falls back to fetching the homepage and following the link that looks like stock, which means it works on a structure nobody has catalogued rather than only on ones already in the list.

It also improved the diagnosis where it did not find a page: essendonmg was reported as a 404 before and is correctly a WAF challenge now. That is the difference between "try another URL" and "never come back", and it belongs in robots_notes rather than in someone's memory.

Two more paths added to the guess list from what the failures revealed, since each miss costs a real fetch and the list is cheap to extend.

Probe: a 4xx robots.txt is no rules, not a refusal

07:52 pm AWST · d516523

The probe bailed on any host whose robots.txt was not 200, reporting "no robots.txt" and moving on. That is backwards: a 404 or 400 means no rules exist, which by convention permits crawling. It was rejecting reachable sites on the strictest possible reading of a missing file.

politeFetch enforces the real robots decision on every request regardless, so nothing here loosens what actually gets fetched — it only stops the probe discarding candidates before they are looked at.

Also carries the observed robots status through to the result instead of assuming 200, so the report distinguishes "permissive robots" from "no robots file" from "WAF challenge". Those are three different situations and only one of them means never come back.

Map groundwork: POA centroids and state outlines from the ABS, decisions recorded

07:49 pm AWST · e2ce15b

GeoNames and data.gov.au both serve 'Disallow: /' to unlisted agents — recorded, not fought. geo.abs.gov.au explicitly allows its ArcGIS query endpoints, and the ASGS POA 2021 FeatureServer returns polygon centroids directly, so the map's two assets come from one sanctioned, CC BY 4.0 8 generalised state outlines (~24 KB). Both are static assets committed with their licence inside them; nothing fetches at request time.

Decisions for the session recorded in docs/VISUAL-PLAN.md: SVG everywhere including the map, ABS over GeoNames for centroids, and two audiences kept on separate pages.

Seed nine BMW dealers, record the Lexus network as blocked

07:38 pm AWST · 5cf46b2

scripts/probe-candidates.ts checks a candidate before it is seeded: robots.txt first, then the inventory page, then which extractor recognises it. Nothing reaches the database on a guess, because a source that turns out to be a WAF or an unsupported platform costs a crawl cycle and leaves a permanently broken row on /coverage.

BMW's dealer network runs schema.org markup at /stock, which the existing jsonld extractor already reads. Nine confirmed and seeded across six states: Sydney, Melbourne, Brisbane, Adelaide, Perth, Canberra, Doncaster, Brighton, Newcastle. No new extractor needed.

Lexus is out of reach and is now recorded as such. Every dealer probed — Sydney City, Parramatta, Brisbane, Perth — serves robots.txt at 200 and then a Cloudflare "Just a moment" challenge on its inventory pages, and lexus.com.au's own sitemap does the same. Recorded with the evidence and left alone.

The probe learned something in the process worth keeping: it originally reported a WAF challenge and a wrong URL guess identically as "no recognised inventory page", which sent the first Lexus run hunting for better paths when every one of those sites was serving Cloudflare. It now tells them apart, because one means try another URL and the other means never come back.

Restore the leading zero NT postcodes lost to a numeric field

07:37 pm AWST · f6a68c2

Berrimah is 0828; Dealer Studio publishes the postcode as a JSON number, so it arrived as 828 and stringified without its zero. 96 live listings across two Darwin dealers carried it, and the corpus counted Berrimah as two postcodes. canonicalPostcode() pads at the store choke point, and 0041 repairs the rows written before the fix. No Australian postcode has three digits, so the pad is unambiguous.

Found while grounding the map build: one postcode failed the four-digit shape check, and the map needs postcode → coordinates to be exact.

Add the prompt for the visualisation session

07:29 pm AWST · 882a818

Self-contained, because a fresh session has none of this context: what Spotlot is, what to read and in what order, the Tier 1 scope, and the explicit instruction not to build time-series until the corpus has the weeks behind it.

It carries the discipline rather than just the task list — every chart states its n, thin cohorts refuse to draw, no trend line under six points, confounds stay on the chart. Those are easy to lose in translation from tables to graphics, and they are what makes the current pages worth believing.

Draft the plan for making Spotlot visual

07:19 pm AWST · 27241ba

For review before any of it is built.

The headline finding is a constraint rather than a design: price-over-time is the main ask and cannot be built yet. 16,092 listings have been read three or more times, which sounds like history until you notice every read happened inside two calendar days. A trend line drawn today joins yesterday to today.

So the plan sequences everything that works on a cross-section first — a map of 183 yards across 225 postcodes, per-dealer profiles, price-against-odometer clouds, ranked turn tables — and puts time-series last, where it becomes the payoff instead of the promise. With a table of what the price chart can honestly show at two weeks, six weeks and six months.

Also sets the principles the visual layer has to keep, because charts imply confidence far more readily than tables do: every chart states its n, thin cohorts refuse to draw rather than drawing something jagged, no trend line under six points, and confounds stay on the chart rather than moving to a caption.

Picker: stop offering combinations the corpus does not hold

07:17 pm AWST · 2875aa9

Badges were keyed on make|model and years on make|model, independently, so neither narrowed the other. The picker would offer a 2019 Wildtrak when the corpus holds Wildtraks from 2012 — a question the data cannot answer, presented as though it could, which is the opposite of what the rest of this product does.

Both are now emitted twice, once model-keyed for the first choice and once keyed on the other's value for the second, so they constrain each other whichever order they are picked in. A Ranger offers 16 years; a Ranger Wildtrak offers 15. Each falls back to the model-wide list when the narrower one is empty, so a rare combination never leaves someone with nothing to pick.

The refine row — transmission, drivetrain, fuel — stays keyed on the model. Narrowing it by badge as well was 11,600 extra rows and a third of the payload, for optional controls, in a document that ships inside every /value render. 28,653 rows and 222KB compressed, against 10,458 and 90KB before: the cost of the fix is real and it buys a picker that cannot ask an unanswerable question.

Unlock 31,000 already-discovered listings, and stop splitting 4x4 from AWD

07:10 pm AWST · e9ef249

Two changes aimed at the same thing: covering more of the market with what we already have.

Depth. We have discovered 78,509 listings across existing sources and hold 47,347 of them. The other 31,162 are not undiscovered — they are behind a per-run detail budget of 200 pages. The biggest yards are the worst affected: Tony White Group 815 of 8,807 (9%), Brisbane Cars 1,047 of 6,507 (16%), Country Cars 24%, Booran 19%, John Hughes 24%. Budgets now scale with what a source actually holds, 700 to 1,500 pages a pass, and the per-host delay on those 21 large sources drops from 8s to 4s — still four times more conservative than the Crawl-delay their robots.txt asks for, and still one request at a time.

Drivetrain. 4x4 and AWD are one car described two ways and dealers use both. The corpus holds 476 Subaru Foresters typed "4X4" and 48 typed "AWD"; the Forester has only ever been all-wheel-drive, so asking for one spelling discarded 90% of the evidence. Same on X-Trail (660/4), CX-5 (328/58), Tucson (327/51). Comparable matching now compares drivetrains by equivalence rather than equality.

What is deliberately not merged is two-wheel drive. A front-drive Tucson and an all-wheel one are different cars at different money — 570 against 378 — and that difference is exactly what the heaviest weighting in the engine exists to protect. 15 cases in CI hold that line.

Coverage: show the marketplace we crawl in the table, not the blocked list

06:51 pm AWST · a1e3695

CarsGuide appeared only under "Marketplaces surveyed", beside carsales and Gumtree, which reads as another one we cannot reach — while it was quietly contributing 200+ of the oldest and cheapest cars in the corpus. A page whose first line promises "every source, its latest crawl and its extraction rate" has to show the sources it crawls.

It is now a row like any other, and the list below is retitled to what it actually is: marketplaces surveyed and left alone, each with its reason and survey date. The distinction the page is making is not dealer versus marketplace, it is crawled versus deliberately not.

Also checked, since /coverage is in the walkthrough: no other source looks broken. Jarvis showed 0 extracted, which is the same fast-pass artefact fixed earlier — its three previous deep passes were 24 of 24 — and the rate column already renders that as an em dash rather than 0%, because it measures the parser against pages attempted rather than found.

README: stop saying the project has not been built

06:17 pm AWST · ece7e98

It read "Status: pre-build. Brand direction and plan are set; no application code yet", and listed three brand files as the contents of the repository. The front door described a project that does not exist, in front of 47,000 listings, a live site and an API.

is — DATA-QUALITY.md is called out, because eight defects and the structural check that stops each recurring is the most useful thing in the repository for anyone deciding whether to trust the numbers.

The crawling and security rules are untouched. They were binding at the start and they still bind. What is added under them is what happened in practice: four marketplaces recorded as WAF-blocked and left alone, one crawled because its robots.txt permits it, one excluded because its robots.txt named an AI crawler specifically. A rule with evidence beside it is worth more than the rule alone.

Re-measure the CarsGuide claim now there are 201 listings, not 24

06:10 pm AWST · 617a714

The claim was made on 24 cars, so it was worth re-checking rather than assuming it held. It holds and sharpens:

CarsGuide dealer sites average age 10.8 years 3.0 years median odometer 139,939 km 16,368 km used 98% 55% median ask $19,990 $41,490

The price row was not in the original claim and is the one that matters most. Franchised dealers advertise their showroom, so a corpus built only from them is nearly-new and expensive. This reaches the sub-$20k end they barely touch, which is where a trade-in valuation usually lives — the corpus was answering confidently about $40k cars and thinly about the ones people actually trade.

Numbers updated in the extractor, its test and the demo notes.

A fast pass should not count listings it cannot fetch

06:08 pm AWST · 8abb910

CarsGuide reported 0 extracted from 206 found on a fast pass, which reads as a broken source on /coverage. It is not broken: a fast pass fetches no detail pages, and unlike the dealer platforms this one carries no listing data on its index, so there was nothing for the pass to do.

listings_found counted those discovered URLs anyway, which made the extraction rate 0/206 rather than "nothing applicable". A fast pass now counts only what it could actually observe. Sources that do yield inline listings are unaffected — their detail URLs were already deduplicated against the inline set, so their counts were right.

The misleading historical row is left alone rather than edited to match the new meaning. It is an accurate record of what that pass did, and the next deep pass supersedes it.

Fix the nav overflowing on a phone, and fill the panel that carries the claim

05:51 pm AWST · 6fde72c

Found by opening the site rather than curling it, which had not been done until now: every page had been checked for content and none for appearance.

At 375px the nav ran off the right edge and took the API link with it — present in the markup, unreachable with a thumb. It wraps now.

The yellow panel states the product's whole thesis and had a hole in the middle of it, because justify-between stretched short text to a tall column. Filling it with decoration would have been the wrong fix; it now carries the evidence that the thesis works, which did not exist this morning: a Ford Ranger takes 42 days to sell across 23 sales we watched happen, market-wide it is 38 days, and 46% were discounted first.

The Ranger row is chosen by most observed sales rather than fastest time. The panel is the claim, so it should rest on the sturdiest number available rather than the most striking one — the same reason the demo notes say to open on the Ranger and not the three-day Yaris Cross.

Also verified while there: the four Market tables scroll inside their own containers on a phone rather than pushing the page sideways.

Move the crawl schedule off the contended minutes

05:17 pm AWST · 2183b53

The 02:15 deep pass arrived at 03:55 and the 08:45 fast pass never fired. GitHub queues scheduled workflows behind every other repository's and drops them under load, and :00/:15/:30/:45 are where everyone puts theirs.

Moved to :07 and :23. This does not make cron reliable — nothing does — so OPERATIONS.md now says to check whether GitHub skipped a run before treating it as a fault, and records why the local crontab is worth keeping alongside: between the two, something runs.

Exclude "light truck", found by auditing every body type rather than waiting

04:50 pm AWST · 57f3c5e

The same Isuzu NNR - Tray was in the corpus twice: once with body type "truck" and correctly out of scope, once as "light truck" and counted as a car. A Mitsubishi F alongside it.

Found by checking all 180 distinct body types against the exclusion list instead of waiting for one to look wrong on a page. Only that one variant had slipped through, which is a reassuring result for a list built by adding things as they appeared.

Exact matching stays, deliberately. Matching on a contained word would catch these variants automatically and would also exclude a Land Cruiser the day a dealer types its body as "Land Cruiser Wagon" — a false positive that removes a real car is worse than a false negative that leaves three trucks in a corpus of 47,000. So variants get found by looking, and this is what looking found.

The audit also confirmed two judgements already made: "people mover" (122) and the long-wheelbase "bus" variants are Carnivals and Sprinters, and they stay.

Demo: volunteer the segment test that failed

04:48 pm AWST · 64b7ef6

Utes were expected to turn faster than SUVs. They do not — 71.6 days against 74.0 across 92 and 192 observed sales. Worth saying out loud rather than quietly moving on.

Hatches do look twice as fast as SUVs and that comparison would clear significance, which is exactly why it is not on the site: it was looked at only after the stated test failed, it is one of ten pairwise comparisons among five segments, and hatches are cheaper and younger so segment, price and age are not separated. Same shape as the 60% interstate gap that was fleet composition.

The negative result is better demo material than another chart would be. It is the evidence that what is on screen was held to the same bar.

Stop the delisting check alarming on crawls that are still running

04:19 pm AWST · ae44069

The health monitor reported "7 delisting(s) from an incomplete crawl". All seven were fine. `complete` is only written when a run finishes, so between recording a delisting and the run ending there is no completed covering run, and the check read that gap as a delisting inferred from a partial traversal. It passed again minutes later.

Worth fixing rather than explaining away. This is the check standing between the product and a fabricated sale — the failure that once buried thirteen cars still on the market — and a guard that cries wolf during every crawl is a guard that gets ignored on the day it is right.

An event is now judged only once no run covering it is still in flight. The invariant is unchanged for anything settled: absence from a partial traversal still proves nothing.

Resolve the two cars the review queue was holding

04:17 pm AWST · 4e4adf5

A 1967 Daimler Sovereign and an Omoda 9 — both real cars, both correctly parked rather than guessed at, which is what the queue is for.

Daimler is Jaguar's sister marque and was simply missing from the canonical list. The Omoda is the combined-showroom case: "Omoda Jaecoo" is one dealer selling two marques, resolved by reading the model, and the rule only knew the C5/E5 and J7 shapes. This listing put the marque back into the model as "OMODA9", so the rule now reads that too. An unrecognised model still returns null and waits for a person.

Both queues are empty, which is the state that makes the number on the home page mean something.

Market: what sells fastest, per model

03:49 pm AWST · e8c06e9

40 model cohorts now carry a days-to-turn figure, up from 6 four hours ago, without anyone doing anything — which is the compounding argument as a fact rather than a claim.

across 18 observed sales, a Mazda CX-5 in 38. Nothing publishes this. It needs the moment a listing stopped being for sale joined to the seller's own listing date, so a competitor starting today has an empty table and no way to back-fill it.

The sales count sits beside every median because several rows are four cars. That is a real number and a noisy one, and the alternative — a threshold high enough to feel comfortable — would leave the table empty for weeks while the answer is already useful. The demo notes say to lead with the Ranger rather than the Yaris Cross for the same reason: open on the firmest row, not the most striking one.

Market: how discounting tracks time on market

03:21 pm AWST · 47bf8d7

326 cars watched out of the market, and the gradient is monotonic across five rising to 76% of those that took over four months, with average price moves going 0.3 to 3.8.

The interesting part is what it is not. Discounted cars sold in a median 72 days against 21 for the rest, which reads like discounting failing to work, and that reading is unavailable: a car listed four months has far more opportunity to be repriced than one gone in a week, so exposure alone produces this shape. Presenting it as "discounting does not help" would be the same error as the 60% regional gap that was fleet composition.

The inference that does hold is the reverse one, and it is the one a buyer can says that in those words, and names the confound before anyone else has to.

This is the observation layer producing something genuinely unavailable elsewhere — it needs the seller's price-move history joined to a listing you watched leave the market, which nobody has unless they were already watching.

Record the page-performance measurement, and why the obvious fix is not built

03:17 pm AWST · 2d6d2a1

/market cold is 3.5s and the intuitive fix — precompute the aggregates into a table during the crawl — turns out to be the wrong one. Timed individually, the nine queries run in parallel and cost about 870ms at the slowest, so the other 2.5 seconds is serverless cold start. Precomputing would buy under a second of the three.

Written down in both docs so the next person to look at a slow page reaches for the measurement rather than the assumption, and so nobody builds a snapshot table expecting it to fix this.

Serve stale aggregates while refreshing behind the response

02:50 pm AWST · fa9cdde

/market builds nine corpus-wide aggregates and takes about three seconds cold. Previously the first visitor after the 60s TTL lapsed paid that for everyone else — on a demo, whoever clicks after a minute of talking.

Expiry no longer blocks: once a value exists it is served immediately and the recompute runs behind the response. Warm renders go from 0.53s to 0.28s and no visitor waits on an expired key. Only the first caller on a cold process waits.

Measured honestly, this does not fix cold starts, and the numbers say so: cold 3.5s, warm 0.28s, and a request landing on a fresh serverless instance still pays full price because the cache is per-instance. The real fix is precomputing these aggregates into a table during the crawl chain, which is where they belong — they only change when a crawl writes. That is not a change worth making the night before a demo, so DEMO.md says to warm the pages first and the architectural fix is written down rather than half-done.

9 cache cases in CI, covering the parts that are easy to get wrong: serving a value that was never computed, blanking the page when a refresh fails, caching a failure, and stampeding on a cold key.

Ship on spotlot.io, and surface how fast the market moves

02:21 pm AWST · 8f17e6c

The domain was already registered on the Vercel account with its nameservers pointed correctly, so this only needed attaching to the project: spotlot.io and www.spotlot.io both serve, HTTP redirects to HTTPS, Let's Encrypt certificates issued for both. Docs and the API examples now name the real domain rather than the deployment URL.

Also adds the Market panel the observation layer has been earning all day. Across 223 cars watched out of the market, spanning 30 makes: a median of 38 days to sell, a middle half of 10 to 89 days, and 44% discounted before they sold.

The quartiles are the point. A quarter of cars go in under ten days and a quarter take more than three months, so a single market number describes almost nothing actually sitting on a lot — which is the argument for giving days-to-turn per cohort, and the panel says so rather than leading with an average.

Both figures are the seller's own, and the panel names them as such, because measuring either from our own observation window produced "sells in under a day" and "0% discounted" earlier today.

Take the discount share from the seller's count, not from our window

01:49 pm AWST · ec8351a

The page said 0% of sold cars had been discounted first. Every cohort, every time. The sellers' own records say 44% market-wide, at an average of 1.7 changes each.

The cause is the one caught this morning in another guise: our price_change_count is what we saw between observations, and we watch a listing for roughly a day of a 41-day life. Nearly every discount happened before we arrived.

It is worse than a wrong number because it inverts the advice. "Nothing sells at a discount" tells a dealer to hold price; the truth is that nearly half the cars that sold had moved on price first. Said in front of a dealer who knows their own market, it would discredit everything else on the screen.

Dealer Studio publishes price_changes_count on the listing, and all 141 confirmed sales carry it. That is used now, listings without it count as unknown rather than as "no cut", and the page names whose figure it is.

The corrected numbers are more interesting than the zeros were: a Corolla sells in 25 days with 80% discounted, a BT-50 in 72 days with 17%. The car that moves fast is the one dealers cut.

Demo: the sale count is a backlog being discovered, not a rate

01:47 pm AWST · 6fa7628

It went 4 to 176 in a few hours, which invites the claim that Spotlot detects fifty sales an hour. It does not. Each source is getting its first pass that can recognise a sold listing, so months of accumulated sales surface at once, and the cars say so: 30 to 121 days advertised on average before selling.

Worth stating before someone in the room does the division. Days to turn is unaffected, because it is aged by the seller's own listing date rather than by when we noticed — which is the same distinction that stopped it reporting a third of a day this morning.

Also worth saying out loud: all 176 carry the dealer's own SOLD declaration and not one came from a 404. The rule this replaced would still be at zero.

The local cron runs the same chain as the workflow, not just the crawl

01:19 pm AWST · 56c60ea

cron-crawl.sh crawled and stopped there. The scheduled workflow crawls, sweeps the taxonomy, enriches, recomputes insights, checks integrity and checks migration drift. So the state of the corpus depended on which of the two happened to fire — and the local one fires every six hours whenever this machine is awake, which is most of the time.

The evidence was sitting in the health line: 36,550 vehicles, 36,096 enriched. The gap is what the local path left behind, along with spec values nobody canonicalised and a review queue nobody swept.

Both paths now do the same work. Each step runs even if an earlier one failed, because a crawl that died half-way still left rows wanting a sweep, and the integrity check is most worth running exactly then. Failures are named in cron.log rather than vanishing into a per-run log file.

Demo: refresh the corpus figures, and say to quote the pages instead

01:18 pm AWST · 92aa6c5

Every number in that paragraph moved while it was being written. It now says so, and points at the live counts.

Adds the CarsGuide counterpart to the WAF-blocked list, which is the sharper version of the same principle: the line is what a site allows, not what we could get away with. It earns its place on evidence — 11.8 years and 127,000km average against 3.1 years and 19,000km on dealer sites.

Demo: the two-listing version, where days to turn changes the money

01:17 pm AWST · 97b4a3c

89 confirmed sales now, and six cohorts have a days-to-turn figure with a real

That spread is worth more than the individual numbers, because the buy price already acts on it. A BT-50 at 72 days trips the slow-cohort rule and its "what to pay" prints "72 days to turn > 45 — margin requirement +2.5%"; a Fortuner at 4 days does not. Two 2021 utes at similar kilometres, and a dealer should pay $30,600 for one and $38,400 for the other, with the reasoning on screen rather than asserted.

That is the argument in ninety seconds: the observation layer is not a statistic on a dashboard, it changes the number that matters, and none of it exists without having watched those cars stop being for sale.

Correct the reason for crawling CarsGuide

12:49 pm AWST · 9d3b6b3

The last commit justified it with private sellers. That was an assumption about marketplaces, not an observation, and checking it found no private sampled says "Dealer". The claim is removed rather than softened.

The real reason is better and is measured. CarsGuide's first 24 listings average 11.8 years and 127,311 km and are 100% used. The dealer-website corpus averages 3.1 years and 19,000 km and is 56% used. Nearly four times the age, almost seven times the odometer — which is the stock a trade-in valuation is usually asked about, and the half of the market franchised dealer sites structurally cannot show.

Seller type is still extracted, because the page states it and a misread would be invisible. It simply is not evidence of anything yet.

Add CarsGuide, and canonicalise spec values at the point they are stored

12:46 pm AWST · 9a2e8e8

CarsGuide is the one Australian marketplace that lets an honest crawler in. robots.txt answers 200 and allows * with Crawl-delay 1, disallowing some filter paths and anything with a query string — so only bare paths are ever fetched. carsales, Autotrader, Drive and Gumtree stay recorded and untouched.

It is worth having because of what dealer websites structurally cannot hold. This corpus is 30% new and 14% demo, because franchised dealers advertise their showroom beside their used stock, and it contains no private sellers at all. Every CarsGuide listing states its seller type outright.

First bounded pass: 33 pages, 224 listings found, 24 extracted, 100% parse rate, budget-stopped and correctly marked incomplete.

That pass is also what exposed the larger problem. The spec table renders as one flat run of "Label Value Label Value", so a parser reading past a label stored "Excellent fuel economy approx. 4.5L/100km EV mode for silent" as a transmission. Reading values up to the next known label fixed the extractor — but the reason it reached the database at all is that store.ts never applied the taxonomy to these fields. canonicalTransmission, canonicalFuel and canonicalDrivetrain existed and were called only at read time.

So the whole corpus had been storing whatever each site said: 26 spellings of four drivetrains, "Front Wheel Drive" 12,288 beside "FWD" 114 and "FRONT WHEEL DRIVE" 222, plus "NOT PROVIDED" and "Chain". No number was ever wrong, because valuation and the picker both canonicalise as they read — but storage that is only correct because every reader remembers to normalise is one forgetful reader away from being wrong, and it makes direct analysis of the corpus misleading.

Now canonicalised at ingest and swept in place: drivetrain is four values, fuel six, transmission two. The sweep works by distinct value rather than by row — 26 statements instead of 32,000 — with the vocabulary staying in taxonomy.ts, and it runs after every crawl so it cannot drift back.

Two spellings added on the evidence: FOUR_BY_FOUR and "Continuous Variable".

Operations: replace the budget estimate with the measurement

12:25 pm AWST · dd82c03

A ten-minute budget that stopped only at source boundaries ran 48 minutes. The same mechanism stopping mid-fetch ran 15.2 minutes against a 15-minute budget. Both measured on real scheduled passes, so the monthly figure in this document is now arithmetic on observed behaviour rather than on intent.

The safety property held in the measured run: sixteen sources were cut short at the deadline and recorded why, and being incomplete they inferred no delistings. The six that finished traversing produced two confirmed sales.

Market: say that the segment medians blend new, demo and used

12:19 pm AWST · 50873ef

The segment table reported a median odometer of 9,235 km and a median age of one year, which is true of the corpus and not of the used market anyone is picturing. These are dealer websites and dealers advertise their new and demo stock beside their used cars: 56% used, 30% new, 14% demo.

An unexplained median like that is a fair challenge from anyone who knows the market, and answering it live is much worse than stating it. The table now gives the mix, and gives the used-only figure — 64,180 km — so the blend is visible rather than implied.

It also says the thing that matters commercially: a valuation does not blend them. Condition is an explicit adjustment there, and a new car is weighted down hard as a comparable for a used one.

Keep the false-positive check for the sold marker

12:15 pm AWST · b18413d

The marker decides whether a car is recorded as sold, so the dangerous direction is the one that fires when it should not: a template carrying a "recently sold" panel would turn every live car at that dealer into a fabricated sale, and fabricated sales would then set days to turn.

`--live` samples live listings across every platform and expects silence. Currently 14 checked across all seven platforms — dealer-studio, easycars, imotor, nextjs-embedded, jsonld, easylist, cox-radius — and none fires. It exits non-zero if one ever does.

Network-bound, so not in CI. Run it after touching soldMarker(), and after adding a platform, which is when the assumption most likely breaks.

Demo: rewrap the lines the last edit left ragged

12:09 pm AWST · 07e8994

Days to turn has numbers: 25 days for a Corolla, 31 for a Kluger

12:08 pm AWST · 4de47db

53 sales confirmed, each carrying the dealer's own SOLD title as evidence, and three cohorts have reached the minimum. The demo script stops describing days to turn as missing and leads with it instead.

The two fixes behind it are visible in the same rows. Every one of those 53 listings shows an observed window of 0.18 to 0.33 days against a seller- published age spanning 0 to 152 — so before this morning the page would have reported a third of a day for every cohort in the corpus, and it would have looked like a number rather than a bug.

Adds a one-slide version, because the number carries the argument on its own: a 2021 Corolla takes 25 days to sell and none of them were discounted first, which is not published anywhere and cannot be back-filled by a competitor starting next month. The mechanics are the moat — a sold car does not vanish, it gets retitled, so the sale is only visible to someone already watching that listing.

Make the minute budget bound the pass, not just its source queue

11:48 am AWST · 1f2982a

Measured on the first real unattended run: a ten-minute budget billed nearly thirty. The deadline only stopped the pass taking *new* sources, so the sixteen already in flight ran on to their own completion — each able to fetch 400 detail pages at eight seconds a page.

That is an estimate wrong by three times in the direction that matters: the documented 1,700 minutes a month would really have been over 5,000, against an allowance of 2,000, and it would have stopped mid-month.

The deadline now reaches into the detail loop, so a pass in progress stops fetching rather than only stopping starting. A source cut short is marked incomplete and records why in its run row, which also means it cannot infer a delisting from an inventory it only half traversed — the same rule that has applied to budget-truncated passes all along.

Found by measuring the run rather than trusting the design.

Coverage: say what the source figure counts

11:20 am AWST · cc7fa92

The page read "166 contributing" while the home page read 181 sources. Both are true and they count different things — extraction in the most recent pass versus holding live cars in the corpus — but one word standing for two numbers is a question waiting to be asked in front of an audience.

The label now states its own basis, and a line explains why it sits below the home page figure: a source whose latest pass was budget-truncated still has its inventory, it just added nothing new that time.

A flag was being read as a source id, and the run went green anyway

11:17 am AWST · 9783a26

The first genuinely unattended crawl fetched nothing. The workflow reported success, every step passed, and the database gained not one row.

crawl.ts filtered its arguments by listing the known flags — `--force` and `--fast` — so `--minutes=12`, added yesterday for the budget, fell through as a source id. It matched no source, the pass had nothing to crawl, and it returned 0.

Both halves are fixed. Anything flag-shaped is now an option rather than a source, so the next flag cannot repeat this. And matching no source exits mistyped id or a flag misread as one, and reporting success for a run that fetched nothing is the worst available failure — the schedule looks healthy precisely while it is doing nothing at all.

Which is the second time today that exact shape has appeared: the scheduled crawls were already finishing in ninety seconds with green ticks because a local pass held the in-flight guard. A crawl that does nothing has now failed loudly twice for two unrelated reasons, so the check is worth having.

Found by running the unattended path instead of assuming it worked.

Age days-to-turn by the seller's listing date, not by our watching

10:50 am AWST · d7f8d23

Caught before the first delisting landed rather than after, which matters, because this one would not have looked wrong.

days_on_market runs from first_seen_at — when *we* first saw the listing. A car we met yesterday and watched sell today reads as one day even if the dealer had been advertising it for two months. Our observation window currently averages 0.27 days. The dealers' own published listing age across the same corpus has a median of 55 days and a mean of 90.

So the moment delistings began arriving, the product's headline number would have read "under a day to turn": precise, confident, and wrong by a factor of two hundred. Nothing about it would have invited a second look — it would have been the fastest-selling market in the world.

Dealer Studio publishes days_old on the listing document and 79% of the corpus carries it, so the last value seen before a delisting is the listing's true age. That is used in preference, and a delisting without it is left out of the cohort rather than mixed in: averaging a 55-day truth against a 0.3-day artefact produces a number that is neither. The basis line on screen says the age came from the seller.

11 cases in CI, covering string and numeric forms, a genuine zero, the 1,075-day listing that really is in the corpus, and the absent and implausible cases that must return null rather than quietly falling back to our window.

Assert that out-of-scope state is state the rules would produce

10:23 am AWST · bc75152

The home page said 11 vehicle names were queued for taxonomy review "rather than silently guessed". All eleven were Adria, BMW Motorrad, Jayco, JB Caravans, Joylong, Kymco, Lvtong, New Age Caravans, Nova, Snowy River and Triumph — caravan, motorcycle and bus marques whose status was decided. An out-of-scope make never canonicalises, so it sat pending forever, which is the failure migration 0016 described for vehicles occurring in the queue that never got the same treatment. The alias sweep now marks them, and the queue is empty, so the number on the page means what it says.

Checking that turned up something worse in kind. Out-of-scope state has been set by hand twice today — a one-off statement covering seven caravans, and the correction restoring 58 vans — and a hand-set row is invisible: it holds until something re-derives it and then silently flips. A Joylong minibus was sitting exactly there, excluded by an UPDATE that no current rule reproduced, because `bus` had since been removed from the body-type list. One row today; the same shape as the thirteen missing migrations.

So the rule now says it — Joylong builds minibuses and light commercial only, named as a make because body type cannot carry it — and check-integrity asserts every out-of-scope vehicle is one the rules would exclude again. State that only a past UPDATE explains now fails the check instead of waiting to be noticed.

Also retitles the delisting check, which no longer only accepts a 404.

Demo: read the price-change count off the page, not this file

10:18 am AWST · 96c32f4

It was quoted as 89 and was 110 within the hour. The corroboration story behind it is fixed and worth telling; the count is not.

Record the delisting finding, and what makes it different

10:18 am AWST · 41ef595

DATA-QUALITY gains entry 8 and a note at the top that it is the one to read. The other seven are wrong numbers, which at least invite scrutiny by being there. This one was an absent number carrying a standing explanation — "days to turn needs more observation time" — that was plausible, comfortable and false. Nothing about a missing number asks you to check it, which is why it survived a full night while every other defect was caught within hours.

The demo script now tells it as a story rather than an apology, and still says the page will show no value until there genuinely is one.

Verified on live pages, not only fixtures: the sold MG HS at Werribee Kia returns a title marker, and the Lismore Toyota MG3 that also left its sitemap returns none and stays live.

An index that yields nothing is not a complete traversal

10:05 am AWST · 83ae95e

complete was set from `!truncated` alone, so a source whose index returned no listings at all counted as fully traversed — which would make every live listing there a delisting candidate in one run.

No source has actually done this; every failing index so far also failed to finish and was already marked incomplete. The guard is here because of the size of the mistake it prevents: burying a dealer's whole inventory as sold. It is the same principle as requiring confirmation per listing — absence only means something if the traversal saw the lot.

Also adds scripts/probe-sold-markers.ts, which is how the delisting rule was index and prints what their pages say. Worth keeping, because the answer is per-platform and the next platform will differ again. It already shows that: Dealer Studio retitles to SOLD, while iMotor leaves "for sale" in the title of a listing that has left the index, so iMotor sales stay unconfirmed rather than guessed.

Confirm a sale from the dealer saying SOLD, not only from a 404

10:03 am AWST · fc136cf

Days to turn is the number this product exists to produce and it had nothing in it. The stated reason was that confirmed delistings need more observation time. That was wrong, and more time would not have fixed it.

The rule required a listing to answer 404 or 410. Across 544 complete traversals it confirmed zero, while every single check logged candidates that had left the inventory index and "still answered 200". Those candidates were not still for sale. Fetching them shows what actually happens:

SOLD 2021 MG HS Essence X SAS23 in Red | Used SUV | Stock #UK15038 SOLD 2023 Toyota RAV4 GX MXAA52R in Silver | Used SUV | Stock #UK15164 SOLD 2022 Tesla Model Y Rear-Wheel Drive in SILVER | Stock #157636

Dealer Studio — 123 sources, 85% of the corpus — keeps a sold car's page up and retitles it. So the rule was discarding the best evidence available and holding out for a weaker one: a 404 is ambiguous, a moved page or a broken link or a migration, whereas a title beginning SOLD is the seller stating the outcome.

Both now confirm, and the event records which did, so any call can be re-checked later.

What has not changed is that a delisting is never inferred. Absence from an index still proves nothing by itself — that is what once buried thirteen cars that were all still for sale — and lismoretoyota shows why the distinction has to hold: a listing left its sitemap with an ordinary title and no sold marker, and it stays live.

The matching is deliberately narrow, because a false positive invents a sale. Only a title *opening* with SOLD, or a phrase that cannot mean anything else. 15 cases cover the real titles and the traps: "Sold Cars", "SOLD OUT", "Soldiers Point Motors", "over 3,000 cars sold", "Sold By:", and the phrase appearing only inside a script tag. In CI.

Also fixes markDelistings selecting live listings unpaged. Several sources hold close to a thousand, and past PostgREST's 1000-row cap a listing was invisible to this check and could never be marked sold however long it had been gone. Third appearance of that cap today.

Merge model strings that name the same car

09:25 am AWST · 8969f61

Two ways one vehicle became two cohorts.

Case. "Santa Fe" and "Santa FE" were separate rows in the picker, splitting 349 Hyundais; likewise Ute/UTE, Ora/ORA, ix35/iX35/IX35, bZ4X/BZ4X. Valuation matches with ilike so its comparables were never split — this was the picker showing one car twice with the count divided.

Normalised to the majority spelling rather than title case, which matters: Hyundai writes ix35 and Toyota writes bZ4X, and initcap would have produced Ix35 and Bz4x, spellings no dealer uses. Whatever most dealers wrote wins, which lands on the manufacturer's styling because that is what they copy. 44 rows moved; every case duplicate is gone.

Sub-brands. GWM sells Haval, Tank and Ora here and dealers disagree about where the sub-brand goes: 440 listings said "Haval H6" and 72 said "H6", 514 said "Haval Jolion" and 36 said "Jolion". Those are the same car and case folding cannot reach them, so a valuation for one genuinely never saw the other's comparables. Now Haval H6 is 506 and Haval Jolion 544.

Written out as an explicit map rather than inferred. A general "strip a leading sub-brand token" rule would also turn "Aion V" into "V" and "Omoda 9" into "9" — those are the models' actual names and useless as labels.

Applied at ingest and in the recurring sweep, for the reason the last commit one-off backfill stops being true the moment it finishes.

Sweep out-of-scope vehicles after every crawl, not once by hand

09:21 am AWST · 2f43386

Classifying at ingest only covers rows arriving after a deploy. The crawl already running carries the older code, so two Hino trucks reached the make picker minutes after the backfill that was supposed to have cleared them — which is how a one-off SQL fix quietly stops being true.

recanonicalise.ts now sweeps first and runs on every scheduled crawl, before enrichment so nothing out of scope gets enriched.

Deliberately set-based in SQL. Sweeping from the client means selecting every mapped vehicle, and PostgREST caps a response at 1000 rows: against 32,000 vehicles that would have swept three per cent of them and printed a success line. The same cap was undercounting contributing sources in health.ts earlier today, so this is the second appearance of one bug.

The vocabulary stays in taxonomy.ts and is passed into the function, so there is one list rather than a TypeScript copy and a SQL copy drifting apart.

Catalogue is 80 makes and none of them a motorcycle, caravan or truck.

Catch motorcycles filed as "BMW Motorrad" with body type "bike"

09:17 am AWST · c0add08

The review queue surfaced a 2026 BMW Motorrad R 1300 GS Adventure. Neither half of the new classification caught it: "bike" is not a spelling the body list held, and "BMW Motorrad" is not a make it held either.

Both closed, along with the nearby spellings dealers use — motor cycle, road bike, dirt bike, trail bike — and the motorcycle divisions of makers whose cars are in scope. Those are named explicitly and never the parent: BMW Motorrad is out, BMW is 469 cars and stays. Suzuki keeps its 1,117 and Honda its 563.

This is the shape of the remaining work here — the classification is right and the vocabulary is incomplete, so the review queue is doing its job by handing back exactly the cases it has not seen before.

Bring the demo script up to the corpus it now describes

08:51 am AWST · df0b663

36,900+ listings across all 181 enabled sources, 80 makes and every one a car, with the field-coverage figures that answer "is this data actually usable".

Price changes move from an apology to a demonstration: 89 verified, and the lag evidence is the interesting part rather than the caveat — 41 of the first 59 carry the dealer's own change date and we saw them 9.9 hours later, which both corroborates them and says something about the market nobody else is positioned to say.

Adds the refresh-rate ceiling to the honesty list. It is the one limitation that bears on the central claim, and it is a billing constraint rather than a technical one, so it should be said out loud rather than discovered.

Exclude motorcycles, caravans and trucks from a car valuation corpus

08:48 am AWST · 7a20b83

The make picker offered Yamaha, Harley-Davidson, Ducati, Jayco and Hino. Valuing a Yamaha would have drawn its comparables from an MT-10, a Niken and a YZF-R7.

The mechanism to prevent that already existed on paper, in three halves that were never joined. isNonCarMake() was defined and called from nowhere. taxonomy_status = 'ignored' was defined and documented in migration 0016, and written only by a one-off statement covering seven rows. Nothing anywhere filtered on it, so even those seven caravans stayed in the picker, the market page and the comparable pool. Motorcycles were never classified at all, because the list only covered caravans.

Classification now runs at ingest, on make and on body type, and listing_current excludes ignored — one choke point rather than a rule the catalogue, market and valuation surfaces each have to remember. The listing, vehicle and observation tables keep every row: what we observed and what counts as a car are separate questions.

Body type is what makes this work. BMW and Honda both sell motorcycles here, so their bikes are excluded while their cars, 465 and 546 of them, are untouched. No make-level rule can draw that line.

Also recorded: the first cut excluded body type `bus` and took out 58 vehicles — HiAces, Transits, Sprinters, a Renault Master, a VW Kombi. Australian dealers file people-mover vans under `bus`, and those are cars a dealer trades; the same change's own comment said vans were being kept. Reverted after reading the excluded list instead of trusting the count.

70 vehicles are out of scope: 29 trucks, 24 motorcycles, 15 caravans, 2 golf carts. The picker is 80 makes, all of them cars.

Recover thirteen migrations that existed only in the database

08:38 am AWST · 9444600

The repository could not rebuild the database it describes. Thirteen migrations had been applied through the Supabase API during the build and never written to the tree, and a committed 0013_integrity_checks.sql carried a different formulation of the cross-view check than the one actually running.

Two of the missing ones are the defences DATA-QUALITY.md points at: delisting_must_be_confirmed, which stops a delisting being inferred from a lagging sitemap, and external_id_not_unique, which stopped a Patrol's price being written onto an X-Trail. A rebuild from this tree would have come up without them, still passing every check, because the checks all run against the live database rather than against a replay of the tree. That is precisely why nothing caught it.

All 28 are now recovered verbatim from the applied statements, numbered in the order the database applied them, with their original comments intact.

scripts/sync-migrations.ts reports and reconciles the difference, comparing content rather than filenames — 0013 was present the whole time and wrong, so a name check would have called it fine. It runs on every scheduled crawl.

applied_migrations() replaces the throwaway export function: security definer because supabase_migrations is not otherwise readable, granted to service_role alone, which can already read anything.

Budget the scheduled crawl so the unattended path actually runs

08:30 am AWST · 9ee73f0

Every scheduled Crawl run for the past day finished in about ninety seconds. They were not crawling: a local pass was in flight, so the in-flight guard made each one exit immediately. Correct behaviour, and it also means the unattended path — the whole point of the schedule — has never completed a pass while looking green the entire time. OPERATIONS.md now says to check the duration, not the tick.

Fixing that exposed the real constraint. This repository is private, so Actions minutes are billed against 2,000/month, and an unbudgeted deep pass over ~190 sources runs about four hours. Two deep and six fast passes a day is somewhere past 10,000 minutes a month; it would have run out mid-month and stopped, which is a worse failure than being slow because it looks like nothing is wrong.

So crawl.ts takes --minutes. A budgeted pass stops taking new sources at the deadline and lets those in flight finish, so every source it touched closes cleanly rather than being killed mid-traversal, and never-crawled-first ordering means the next pass resumes exactly where this one stopped. The schedule now spends ~56 min/day, about 1,700 a month, with headroom for CI.

The cost is freshness and it should not be buried: a full sweep now takes several days rather than four hours, and price-change detection is a direct function of re-read frequency. OPERATIONS.md sets out the three ways to lift it — public repo, any always-on host, or chunked pg_cron invocations — since each is a decision rather than an operational tweak.

datasets that move weekly at best. Tied to the deep passes instead.

Manual dispatch gains minutes and fast inputs, so the unattended path can be verified for eight billed minutes instead of four hours.

Refuse an asking price that is the car's own stock number

08:25 am AWST · 69cb713

The health check flagged a 2025 Mazda CX-60 at $11,293,498. The dealer's site genuinely publishes that as its price, alongside stocknum "11293498": their CMS had filled the price field with the identifier. The parser read the right field; the source was wrong.

The range bound caught it by luck, because the stock number ran to eight digits. The same fault on stock number 45990 yields an ordinary $45,990 that no bound can distinguish from a real ask, and it would have moved a valuation silently. So the new rule is provenance rather than magnitude: a price equal to its own car's identifier is refused whatever its size.

Vetting happens in credibleAskingPrice(), at the single point every extractor passes through, so no future extractor can reintroduce the class. It compares against the identifiers the site publishes, not the one parsed from the URL — this car's external id was "60-11293498", which does not equal the price, so URL-derived matching would have missed it. Refusal governs only the typed column; the raw payload still holds what the site said.

Zero false positives across 59,837 priced observations: it matches exactly the one car. A real Hiace asking $44,500 under stock number 544500 is untouched, which is why the test is equality and not a substring.

The offending price is nulled, keeping the observation and its raw fields. No insight run had used it — the last one preceded it by five minutes — so nothing downstream needed retracting.

The guard applies from the next crawl invocation; the pass running now loaded the earlier code.

- credibleAskingPrice() + wiring in observeListing - check_price_equals_identifier() asserts it in SQL (migration 0015) - implausible prices upgraded from warning to failure, now that storing one means the guard was bypassed rather than that a dealer published something odd - scripts/test-price-guard.ts, 15 cases, in CI - DATA-QUALITY.md defect 6, and why this class cannot be caught by inspection

Store an unchanged field set once, not on every observation

07:18 am AWST · c9ac8f7

listing_observation is append-only and meant to be kept forever, so its growth is the one that compounds: four passes a day over 60,000 listings is a quarter of a million rows daily, and at 2.7KB each that is around 650MB a day indefinitely. The database is at 2GB after one night.

The mutable facts — price, odometer, photo count, description length — have their own columns and are always recorded. `fields` is everything else the parser found, and on a car that has not changed it is byte-for-byte what the previous observation already holds.

Each observation now carries a hash of its field set. An unchanged hash means the payload is stored empty, and the value at any moment is the most recent non-empty payload at or before it. History stays complete, and the raw capture still holds the page.

The first attempt at this was wrong in a way worth recording: it compared the new field set against the previous observation's stored payload, so once one observation stored the "unchanged" marker every subsequent comparison failed against the marker and re-stored everything. Hashing the content rather than comparing to whatever was last written fixes that, and a resolver in raw.ts is the single read path — the same discipline the deduplicated raw bodies needed, because a reader taking the empty payload at face value would silently see nothing.

Kawasaki is a real marque

06:46 am AWST · a05aef6

Record which sources are deliberately disabled, and that re-seeding re-enables them

06:24 am AWST · ec2ce5a

The seed scripts upsert enabled: true, so re-running one would silently switch a source back on that was turned off for cause — a 403, or a site whose stock sitemap lists category pages rather than vehicles. The reasons live in source.robots_notes, which is the durable record; this is the pointer to it and the warning about the footgun.

Correct for choosing the group after looking at it

05:48 am AWST · 46cbf30

The insight engine surfaced "Gold cars ask 11% above what their age and kilometres predict, p = 0.023" on 52 cars. It should not have. Both group insights pick an extreme — the dearest state, the dearest colour — from many candidates and then test that one. Testing the winner of thirteen colours at p < 0.05 is not a 5% false-positive rate, it is roughly thirteen times that, because the maximum of thirteen noisy groups is extreme by construction.

This is the exact failure the module was written to prevent, and I built it in the selection uncontrolled. The p-value is now multiplied by the number of candidates that could have produced the claim — Bonferroni, conservative, and easy to state on screen, which matters more than squeezing significance out of a marginal finding. The workings name how many colours or state pairings it could have been.

Under the corrected bar nothing currently clears, including the QLD/SA spread reported earlier. That is the honest position: it was a 7% difference selected as the most extreme of 28 possible state pairings.

Stop storing page furniture on every observation

05:19 am AWST · 2c13576

At 8.5KB an observation and 43,000 observations, listing_observation had reached 457MB and the database 1.5GB. Most of that was not facts about a car. The largest keys were call-to-action buttons, finance lender lists, image and thumbnail URL arrays, the site's marketing blurb, a duplicate of the JSON-LD whose values are already parsed into columns, and features stored three times over — in their own column, raw, and grouped.

SPEC asks for every field a listing carries, and that promise is kept by the raw capture, which holds the whole page for re-parsing. `fields` is the parsed, queryable layer, and it does not need a second copy of the page. README is also explicit that we store facts and a link back rather than republishing photography or description text, so image arrays and site blurb did not belong there at any size.

Average field size halves, 8,527 bytes to 4,184, and the same pruning is applied to what was already stored — recoverable from the captures if ever needed. Growth of the fastest-growing table is halved with no analytical loss.

Demo script and taxonomy refresh at 23,000 listings

04:47 am AWST · 9621c8b

Sub-brands published as makes now resolve: TRD is Toyota Racing Development, so a TRD Hilux is a Toyota with TRD in the badge the listing already carries. Same for AMG and BMW M. Denza, Forthing, Tata, MV Agusta and BMC are real marques and are added. Review queue back to zero.

Demo script rewritten against the corpus as it actually is — 23,000 listings, 148 contributing sources, 93 makes, 92% with a VIN — and points at the data-quality record when asked how we know the numbers are right. That document is a better answer than claiming nothing went wrong.

One round trip each for the catalogue and the coverage counts

04:30 am AWST · d1b83c2

Paging an RPC re-runs the whole function per page, so the catalogue's 10,458 option rows meant eleven executions of the same aggregate on every uncached request — most of what was left of /market's four seconds. It now returns a single JSON document. The eleven separate exact counts behind every page became one call for the same reason.

Build the picker options in SQL too

04:23 am AWST · 9ee85c8

The valuation form's make/model/badge/year/spec lists were assembled by paging every live listing and grouping in JavaScript — the same shape as the market tables, and the same cost. As aggregates they are a few thousand rows, so the database produces them directly and the canonicalisation of spec values happens on the way out, which keeps "4x4", "4X4" and "4X4 Dual Range" one option rather than three.

With that and the earlier aggregate work, /market is 0.74s and /value 0.04s locally, against 8.7s and 2.5s before. The cost no longer scales with the corpus, which matters because the corpus has grown fivefold tonight and is still growing.

Move market aggregates into SQL and insights into the pipeline: 8.7s to 0.66s

04:19 am AWST · 3ec10a9

/market took 8.7 seconds cold at 20,000 listings, and the cost scaled with the corpus — it would have been unusable by the time the corpus doubled, and the first visitor after any cache expiry paid it.

Two causes, both self-inflicted. The segment, make and state tables paged every live listing over the network to take medians in JavaScript; those are now single indexed queries in Postgres. And the insight engine re-fit every cohort's price-by-age-and-odometer curve on each request, which needs the whole corpus — that now runs once per crawl and stores its result, so the page reads what the pipeline last established along with the corpus size and time behind it. An insight that stops clearing the bar is deleted rather than left on screen.

Crawl never-crawled sources first: 88 of 183 had never been visited

03:47 am AWST · 8b6a485

A pass takes hours and keeps getting interrupted — a fix needs deploying, a laptop sleeps — and sources were processed in a fixed database order. So every restart re-crawled the same early sources while the tail never ran at all. Eighty-eight of 183 sources had never been visited once, including nearly all of those discovered tonight, which is why the contributing count sat at 78 across three health checks while the corpus kept growing.

Sources are now ordered never-crawled first, then stalest, and the run prints how many have never been visited. It is the rotation already used for detail pages within a source, applied a level up, so an interruption costs the least-covered sources nothing rather than everything.

Write down the five data defects and their defences

03:19 am AWST · b1cf464

Every one produced data that was internally consistent, correctly typed and wrong, and none would have been caught by a schema or a unit test on a fixture. What caught them was noticing a number was not plausible and going to the source. That should not be the control, so this records what each was, what caused it, and which guard now prevents it — and says plainly where the guard is a structural check versus where it still depends on someone looking.

Weight comparables by condition: new stock is a different market

03:18 am AWST · da5748e

The corpus is 9,378 used listings averaging $41,522, 4,412 new averaging $55,437 and 1,729 demo averaging $51,670, and the valuation weighted all of them equally. New-car pricing is recommended retail, factory incentives and dealer margin — not a used price with fewer kilometres on it — so a used valuation drawing on new listings is comparing across markets.

Condition is now part of the subject, defaulting to used because this is a trade valuation and the subject is a car someone owns. Comparables in a different condition are down-weighted, hardest between new and used, and the comparables table shows each car's condition so the mix is visible.

The effect is smaller than the raw averages suggest — $41,879 against $42,743 for a 2021 Ranger at 80,000km — because the odometer adjustment was already pulling zero-kilometre cars most of the way down. The weighting still belongs makes the assumption visible instead of implicit.

Clear the review queue: motorcycle marques in, golf carts and caravans out

03:16 am AWST · 0dc99e8

Ducati, Hyosung and Vespa are real marques a dealer listed. Lvtong builds golf carts and New Age builds caravans; both are recognised and out of scope for a car product rather than unknown. Motorcycles stay in the corpus and segment as motorcycles, so they are visible as their own category rather than quietly inflating a car figure.

Stock numbers are a label, not a key: stop matching listings on them alone

02:50 am AWST · 541cd60

The X-Trail's $58,917 "gain" was not a parser fault after all. countrycars publishes a 2026 Nissan Patrol and a 2026 Nissan X-Trail both carrying stock number 508390, on separate URLs. Listing matching fell back to the stock number when the URL did not match, so the Patrol's $98,800 was recorded as an observation of the X-Trail. The Navara's $11,110 "drop" was the same thing with 508318, shared with a 2025 X-Trail. Three consecutive reads of the X-Trail page return $39,883 with the right id, which is what ruled the parser out and sent me to the capture log, where two different cars were fetched seconds apart carrying one number.

The schema asserted this could not happen — (source_id, external_id) was unique — which is exactly why the wrong row was found instead of a conflict being raised. A URL is the identity of an advertisement; a stock number is a label the dealer controls and evidently reuses. The index is now non-unique, and matching requires make and model to agree before a stock number is taken to mean the same car.

The extractor separately now refuses a document whose id does not match the URL it came from. That was not this bug, but the same class, and it is cheap

Misattributed observations and their events are deleted; eleven verified price changes remain.

Make implausible price movements prove themselves

02:47 am AWST · fe3fb8c

Three separate causes of fabricated data turned up tonight — diffing across two views, inferring a delisting from a stale sitemap, and reading a second vehicle's document from the same page. Each was caught by noticing the number was not credible and going to check. That should not depend on someone looking.

A price change beyond 30% of the previous price, or $20,000, now has to be corroborated by the listing page before it is written. If the page disagrees, or cannot be reached, no event is recorded: losing a genuine repricing costs one crawl cycle, while inventing one corrupts the price history the product rests on. Rejections are counted and reported per source, because a source rejecting many is a parser problem rather than a quiet market — a silent drop would just hide the next bug.

This would have caught the X-Trail's $58,917 "gain" on its own.

Read the right car: match the vehicle document to the URL it was fetched from

02:19 am AWST · 0f29e84

A 2026 X-Trail appeared to gain $58,917 in three hours, and a Navara beside it to lose $11,110. Neither happened. Both pages carry more than one carSSR vehicle document, and the extractor took whichever appeared first, so a second render attributed another car's price to this listing. The site settles it: the X-Trail reports price_changes_count 0 and last_price_change_at four months ago, and the page today contains no occurrence of $98,800 at all.

The URL ends with the vehicle's id, which the extractor already parses for externalId, so the document is now selected by matching that id. Where several candidates exist and none is identifiable, extraction returns null rather than the price history the whole product rests on. Verified against both live pages — $39,883 and $58,990, each with an externalId matching its URL — and the fixture suite still reports 100% make/model and 99.6% price across 231 listings, so nothing regressed.

The two fabricated events and the misattributed observations behind them are deleted. An identically-priced GWM at another dealer was left alone.

Resolve dealership names published as makes; review queue reaches zero

01:49 am AWST · 218f3e0

"Omoda Jaecoo" had grown to 14 vehicles. It is not a marque — it is one showroom selling Chery's two brands, and picking either would have been inventing the answer, which is why it sat in the queue rather than being guessed. The listings settle it: every one is a J5, J7 or J8, which is Jaecoo's range, while Omoda sells the C5, E5, C7 and C9. Make resolution can now consult the model for names like this, giving 14 Jaecoo and 2 Omoda — decided by the data. A model matching neither pattern still goes to review.

Also from the queue, all verified as real marques now selling here: Farizon (Geely's commercial EV arm, five SuperVans), Alpina, Yamaha and Indian. JB Caravans and a Nova Terra Sportz whose body type is "2 axle" are caravans and are marked out of scope.

The review queue is empty: 10,551 mapped, 4 deliberately ignored, none pending. That is the loop working as designed — every entry was surfaced by real listings and each got a decision rather than a default.

Corroborate index-detected price changes against the listing itself

01:47 am AWST · 8338b7a

Seven more price cuts arrived tonight, all from one dealer group and all between $1,000 and $5,000. Verified rather than assumed: the largest, a 2022 Landcruiser cut $103,990 to $98,990, agrees with its live listing page at $98,990 and the site reports price_changes_count 3. A dealer group repriced its stock and the search index caught up overnight — a real market event.

The verification exposed a gap. Those events carry no site_reported_at, because the index document does not publish last_price_change_at; only the listing page does. So the timestamp is missing exactly where it matters most, on changes detected through the view that lags — the Landcruiser was cut at 03:59Z and seen at 17:24Z, thirteen and a half hours later.

A price change is rare and expensive to get wrong, so it now earns one fetch of the listing itself, the same discipline delistings got. That confirms the new price against the authoritative page and recovers the platform's own timestamp for when the change actually happened. Best-effort throughout: every failure path returns null and the change is still recorded, because failing to confirm must never lose the observation.

Delistings must be confirmed by fetching the listing, not inferred from absence

01:19 am AWST · d305d93

Thirteen cars were marked delisted tonight. All thirteen were wrong: every one of their listing pages still answered HTTP 200. They had dropped out of the dealer's sitemap, which lags exactly as its search index does, and the crawler read absence from that cache as absence from the market. The pattern gave it away before the check did — thirteen cars vanishing inside three and a half hours from one dealer, including brand-new 2026 stock, most without a price ever recorded.

This is the same mistake as the fabricated price changes, one level up: a stale view treated as a real-world event. And it is the worst place to make it, because a listing that disappears without a price cut is the strongest free signal that a car sold — the differentiator the whole product rests on.

A candidate is now delisted only when its own page answers 404 or 410. One that answers 200 is still for sale and merely missing from a cache: it stays live, and the run records how many candidates turned out that way. The confirming status is stored on the event, an integrity check refuses any delisting lacking it, and the health monitor alerts on the same.

The thirteen are deleted and their listings restored to live.

First real price movement, and a record of when we saw it versus when it happened

12:48 am AWST · 15c3918

Four price cuts recorded, all verified real rather than assumed. Each is an index-to-index comparison with no cross-view contamination, and the one with no detail observation to corroborate it was checked against its live listing the site's own document reports last_price_change_at 2026-08-10T02:13Z with price_changes_count 12. Genuine.

That timestamp is the finding. The cut happened at 02:13Z and we saw it at 16:38Z, because a dealer site's search index refreshes hours behind its listing pages. Dating a price cut fourteen hours late matters for a product whose claim is watching price movement, so the gap is now recorded instead of platform's own timestamp where it publishes one — kept in its own column rather than overwriting observed_at, which stays strictly the time we saw it. The listing page shows both, and says a change seen through a search index can already be hours old.

Keep raw bytes where they are worth something, not on every fetch

12:35 am AWST · 0fd2dec

raw_capture is 267MB of a 426MB database after one partial pass, and a full deep pass across 183 sources at 200 details each would store about 1.5GB of bodies — twice a day. That would fill the disk within days and buy very little, because the second capture of an unchanged page is already deduplicated by sha, and the tenth capture of a page whose only change is its price adds nothing a re-parse could use that the observation row does not already hold.

SPEC's reason for retaining raw payloads is that a better parser can re-read every past listing without re-crawling, and that needs one good body per listing rather than one per observation. Bytes are now kept for index pages (few, each covering ~20 listings), for anything extraction failed on (the whole point of keeping them), and for a listing's first body (which delivers the re-parse promise). Everything else records the fetch — url, status, sha, size, time — without the payload, so the audit trail of what was requested stays complete.

The detail loop now extracts before storing, so a capture knows whether its bytes are worth keeping: a page the parser choked on is exactly the one worth having later.

27 more verified dealer sources: 156 -> 183

12:33 am AWST · c52a65a

Found by mining vendor and dealer-group directory pages rather than searching dealer by dealer — a group's own /dealerships page enumerates its sub-brand sites and they are nearly all on one platform. QLD 12, VIC 8, NSW 6, WA 2.

Excluded and recorded rather than worked around: four sites behind a WAF, one rate-limited, one with an expired certificate, one whose robots disallows the inventory path, and one whose robots.txt names ClaudeBot specifically. Two more are confirmed on a supported platform but currently list no used stock, so they were left out rather than counted as coverage that does not exist.

The report flagged that two of the EasyCars sites use themed detail paths (/vehicle-list-view/<slug>--<digits>, /our-vehicles/<slug>-<digits>) and suggested the extractor's link matcher might need widening. Tested against all three live: it already handles them, because the matcher inspects only the last path segment and the double hyphen still satisfies it. 10-12 priced listings extracted from each with pagination detected, so no change was made — the suggested fix would have been a change to working code.

Align the remaining iMotor seed budget

12:22 am AWST · 310c72a

Raise detail budgets to what a five-hour pass can actually cover

12:22 am AWST · bd17f3e

43 sources have discovered 22,376 listings and extracted 3,572 — coverage is bounded by how much a pass fetches, not by the parsers. At 16 concurrent hosts and one request per ~9s per host, five hours (the ceiling before the Actions job limit) is roughly 32,000 fetches, or about 200 per source across 156. Budgets go to 200, and rotation covers the remainder of the big sitemaps over subsequent passes.

EasyCars stays at 15: those sites declare Crawl-delay 20, and their index cards already carry price, odometer and identity inline, so detail pages buy the least there per second spent.

Report parsed and covered separately; raise cross-host concurrency to 16

12:21 am AWST · 6d5f5d5

The crawl log showed one percentage, extracted over found, which reads as a broken extractor when it is a per-source budget. Westside discovers all 1,794 cars in its sitemap and fetches the 150 it is budgeted: that is 100% parsed and 8% covered, and I misread my own log line as a parser failure, which is evidence enough that it needed splitting.

Concurrency 8 to 16 for the same reason as before: every source is a different host and politeness is enforced per host, so this adds nothing to any individual site and roughly halves a full pass. Coverage of the large sitemaps is currently limited by how long a pass takes, not by the parsers.

Say "no recall data held" rather than "no recalls matched"

12:19 am AWST · 9295395

The depth panel read "No recalls matched for this model", which sounds like we checked and the car is clear. We hold zero recall rows: vehiclerecalls.gov.au refuses our crawler at the connection level, and the ACCC feed — tested tonight, returns 200 — carries consumer-product recalls, its car-parts category being aftermarket control arms and child-seat adapters rather than manufacturer campaigns. It is not a substitute, and that is now recorded on the source.

Absence of evidence stated as evidence of absence is the same failure as inventing a number, and safety is the worst place to do it.

Overnight loop keeping enrichment level with the corpus

12:16 am AWST · 96a4292

The crawl adds vehicles all night; enrichment only ran when I ran it, so the context layer fell a thousand vehicles behind within half an hour. This tops it up every twenty minutes, processing only what lacks enrichment, and re-runs the taxonomy over anything parked in review.