Every change, in the words it was made with

Health check: the hygiene counts stop scanning the corpus

11:45 pm AWST · a54eadd

The 23:36 monitor pass printed its summary line and then failed:

ALERT health check failed: count of listing_observation could not run

That is the script working. It throws rather than reporting a zero it could not observe — the behaviour added after it once printed live=0 over 152,036 live listings (DATA-QUALITY entry 16). But a monitor that alerts on its own weight trains you to skim past it, and the next alert will be a real one.

The failure is not the plain obs= count, which succeeded on the line above it. It is the two hygiene assertions further down: no present observation missing its source_view label, no present observation priced under $500 or over $2m. Both must return zero, both always have, and at 557,714 observations the cost of proving it was 2,732ms on an index scan and 3,499ms on a parallel seq scan over the entire table.

0080 and 0085 met this class by raising the ceiling, which is right when the query has real work to do. These two do not. The predicate is the violation, so a partial index over it holds only the rows that break the invariant — none, one empty page, 8KB each — and both counts become index-only scans that find nothing: 0.131ms and 0.158ms, one buffer, zero heap fetches.

The point is the shape, not the speedup. A partial index is only ever as big as the problem it looks for, so this stays flat as the corpus grows rather than needing a larger timeout every few weeks. Raising a ceiling buys time; this stops the clock.

Built with CREATE INDEX CONCURRENTLY against the live database — 19 crawl runs were inserting, and a plain CREATE INDEX would have blocked every one of them until the build finished.

Give CI the database its build has always needed

10:50 pm AWST · e4f565a

The web-build job has never passed. Forty runs, every one red, since the commit that added it — which makes it decoration rather than a check, and a red line that always means nothing is worse than no red line at all. It is the same failure every time:

Error: coverage counts failed: TypeError: fetch failed Export encountered an error on /page: /, exiting the build.

The placeholder credentials satisfied db()'s "must be set" throw and nothing past it. `/` is prerendered with revalidate 60, so building it calls coverage_counts() for real, and example.supabase.co does not answer.

The two ways to keep the build database-free are both worse. Making `/` dynamic drops the static render on the page this product is judged on. Letting the prerender fall back to placeholder numbers would ship a home page quoting a corpus that does not exist — and the throw is already the right behaviour in production, where a failed ISR regeneration leaves the last good page serving rather than replacing it with zeros.

So the build gets the secrets that already exist in this repository and that crawl.yml already uses, scoped to that one step; every test above it is a pure function and stays that way. The cost is a handful of RPCs per push.

One property worth naming rather than engineering around: the build now depends on the database being responsive, so a push during a heavy crawl can fail it. That is true of a production deploy too — this makes CI honest about it rather than immune to it.

Days to sell comes off the request path

10:36 pm AWST · 5341561

The two panels that carry the pitch were blank on the live site: "median days to sell" and "cut before selling", both from sale_speed(), which reads the seller-published `fields` of every delisted listing. Measured 5.6s against PostgREST's 8-second ceiling, on a page that fires ten queries at once while 24 crawl runs are writing. It did not fail every time — it failed whenever the instance was busy, which is when someone is most likely to be looking, and tolerant() then rendered "needs observed sales" over 1,939 confirmed sales. An absent number with a comfortable explanation is entry 8, and this was one.

Not the merge: a `status = 'delisted'` scan never touches a merged row, and the plan is unchanged — still an index scan on listing_status_idx. What changed is the corpus. From EXPLAIN, the 5.6s is 1.7s resolving one observation per delisted listing and 3.8s more resolving the deduped fields payload through the second lateral. Both are per-row TOAST reads and both grow with the delisted set. No index helps.

So it joins every other heavy aggregate on a board, refreshed by the 6-hourly job that already exists under a 15-minute ceiling. sale_speed() keeps its signature and reads the board: 6,703ms to 112ms, same numbers — 1,939 sales, 53 makes, median 32 days, middle half 10–72, 47% cut before selling. check_stale_boards() gains the new board, so a refresh that stops is a failed run rather than a panel claiming there are no sales.

market_pulse() is the same lesson one step down. It reads delisted_at and price only, but took them from listing_current, which always pays for the fields lateral because a LATERAL with LIMIT cannot be join-removed — the exact reason 0060 created listing_current_lean. Moving it to the lean view takes that scan from 2.2s to 1.56s for nothing but the right view name.

Verified on the live site: all four panels populated, 12/12 integrity checks pass, all boards fresh, tsc clean, and the tree rebuilds the database exactly.

A check that cannot run must not print ok

10:36 pm AWST · 94c8ecf

Found while answering "did we break anything" after the merge: health.ts reported live=0 over a corpus of 152,036 live listings. Nothing was broken — the exact-count scan was cancelled under crawl load, and `const { count: n } = await q` returned `n ?? 0`. The next two runs printed the real number.

Pulling that thread found the same bug somewhere it matters far more. check-integrity.ts was printing

ok no price equal to the listing's own identifier

for a check that had not run at all. check_price_equals_identifier() reads `fields` for every priced observation — 3.7 million buffer hits, 78 seconds measured — and had crossed the API role's 8-second ceiling as the corpus grew past half a million observations. The RPC returned an error, the caller destructured only `data`, and null counted as zero. The suite that gates the crawl workflow was green because it could not see.

So every check in check-integrity.ts and every count in health.ts now reads its error and throws. Two helpers, rpcCount and rowCount, make that the only way to write one; nine call sites moved onto them.

0085 gives the two slowest check functions their own 60-second ceilings. That is measurably real — the cancellation moved from 8,171ms to 60,223ms, so a function-local statement_timeout does re-arm the timer for the enclosing statement — and it was still not enough for a 78-second query. Filtering to observations that carry a payload does not help either: 188,642 rows instead of 485,568, but they are precisely the big ones and the same 3.7 million buffers come back.

0086 therefore moves that invariant off the request path, to where a long statement is already allowed: pg_cron at 45 past, every six hours, under the 15-minute ceiling 0073 and 0079 established. The invariant is unchanged and still exact over the whole corpus; what changes is that the answer is stored with its computed_at, and the suite fails on a stale row as well as on a non-zero count — 0077's argument, applied to a check rather than a board.

First honest run of it: 0 across 485,798 observations. The invariant holds; we simply had not been asking.

sync-migrations reports the tree rebuilds the database exactly, and the bootstrap cron job used to populate the first row was unscheduled after it did.

Merge the dealerships that were seeded twice

10:36 pm AWST · a391724

132,730 live listing rows resolved to 129,190 URLs when entry 14 of docs/DATA-QUALITY.md was written; by this evening it was 156,142 rows over 151,550. The gap is one dealership seeded under two source ids — a rename (GWM Haval to GWM, SsangYong to KGM), a group prefix, an abbreviation — whose second domain redirects to the first, so both crawls stored the identical URL and every published count included that yard twice.

0078 made the published number distinct URLs, which fixed the arithmetic and not the corpus. This fixes the corpus.

0082 gives the schema the word it was missing. A redundant row is neither deleted — that would take its observations with it, and they are append-only — nor delisted, which would have manufactured 4,596 sales out of a filing error and poisoned the one signal days-to-turn is built on. It takes status = 'merged' and names the row recording the same page, with a constraint making that pairing compulsory both ways: a retirement names its survivor or it is not a retirement. A third status rather than a flag, because forty-odd SQL functions and every page already ask for 'live' or 'delisted' by name, so 'merged' drops out of all of them at once; listing_current excludes it outright for the readers that filter nothing, like the insight corpus, where a duplicate page would vote twice in the price-against-age fit.

0083 is the merge, computing its targets at apply time rather than from a hard-coded list. The surviving row for a URL is the one whose source's base_url is the host the page is served from — a fact already in the data, since the redundant id had been storing absolute URLs on the survivor's host all along. It cuts against taste in both directions: ftgchery beats cheryferntreegully, mandurahgwmhaval beats mandurahhaval. For the 32 URLs where no id matches the host (cheryadelaide and cherynailsworth both redirect to cherymainnorth, which no source is named for) the earliest-observed row survives, which is arbitrary and recorded as such.

Ten live pages only a retired id had seen moved to the surviving id rather than being retired — nothing would ever have looked at them again. Delisted leftovers stayed put, because check_delistings_from_partial_runs judges a delisting against the crawl runs of its own source. Before: 156,142 live rows over 151,550 URLs. After: 151,604 of each.

0084 makes it an invariant — no URL live under more than one source id — and check-integrity.ts runs it on every crawl. Discovery will seed a yard twice again; it now costs a failed run the next morning instead of three days of publishing a number 3% too big.

retired row; the home page's cut tape drops events whose listing it cannot describe instead of rendering a price against a blank car; and the three surfaces where a retired id is still reachable say what happened rather than reading as a dealer with empty lots.

What the merge does not fix is in entry 14 with counts: 192 duplicate vehicle rows, 42 of 6,620 price-change events still counted twice by market_pulse and model_movers, and 8,701 observations that belong to retired rows and stay in the total.

clean, sync-migrations reports the tree rebuilds the database exactly.

Wire the checks to something that runs them

09:10 pm AWST · 90f1634

The stale-board check landed in health.ts, and health.ts is run by no workflow at all. Its alerts were firing into an empty room, which is defect 7's exact shape: a rule that existed and was never called.

So the board assertion also goes into check-integrity.ts, which the crawl workflow does run and which fails the run when it trips. A board whose refresh has stopped puts an empty market on screen over 131,000 cars, and data that is wrong by omission belongs with the other invariants rather than only in an operational one-liner.

health.ts is now a workflow step too, after integrity, so the quieter signals it carries reach somebody.

Verified both paths by making them fail: market_make_board backdated 18 hours produced "FAIL every precomputed board is fresh" and exit 1 from check-integrity, market_state_board backdated 20 hours produced the ALERT and exit 1 from health, and both timestamps were restored to the values they held before.

Compute the latest run per source in the database

09:09 pm AWST · 6a672ee

listings.

Three causes, each hiding the next.

/coverage took the newest crawl_run per source regardless of whether it had finished, and an in-flight run's counters are zero until it writes stats at the end — so with 24 sources in flight at any moment, roughly two dozen rows read zero at all times.

Reading the newest *finished* run instead was not enough: a run abandoned when a crawler is stopped has finished_at set so the lock clears, and counters still at zero. Twenty-four such rows were created at 18:32. The check became finished *and* it did something.

That still left dashes, because the page fetched every crawl_run and reduced in TypeScript. There are 4,244 runs, PostgREST caps a response at 1,000 rows server-side, and 1,397 runs are newer than Westside's last good one — so it was never in the data the page received. Asking for limit(8000) changed nothing; the ceiling is the server's, not the request's.

latest_source_runs() now returns one row per source, computed in the database: the newest run for whether a crawl is running, and the newest run that finished and did something for the numbers. ~700 rows instead of thousands, and no cap to outgrow.

Fourth time today an unbounded PostgREST read truncated at 1,000 without saying so — after probe-sitemaps' source query, the insights rotation, and the vehicle lookup in the basis evaluation. The pattern is in DATA-QUALITY entry 16; this is the variant where the fix is to stop fetching rows you do not need.

Two more reasons /coverage said zero

09:05 pm AWST · ebdd40c

The first fix — read the last finished run rather than the in-flight one — did not clear Westside, because two other things were wrong.

A run abandoned by a stopped crawler has finished_at set so the lock clears, and counters still at zero because it never reached the point of writing them. Twenty-four rows like that were created at 18:32 when I stopped the crawler to pick up the new rate settings, and "newest finished run" happily selected one. The check is now finished *and* it did something — a page fetched, a listing found or extracted.

And the run query had no limit, so it took PostgREST's default 1,000. One pass over 697 sources writes about 700 rows, so this page could see roughly 1.4 passes into the past; any source crawled before that read "not yet crawled", on the page whose purpose is saying when a source was crawled. Now 8,000.

Third instance today of an unbounded PostgREST read quietly truncating at 1,000, after the sitemap sweep's source query and the insights rotation.

A source being crawled looked like a source finding nothing

09:03 pm AWST · 65972ad

listings and had been crawling for three minutes.

/coverage took the newest crawl_run per source regardless of whether it had finished, and an in-flight run's counters are zero until it writes its stats at the end. So every source currently being crawled read 0 found, 0 extracted, 0 new — on the page whose entire job is to say which sources are working. With 24 sources in flight at any moment, roughly two dozen rows were lying at all times, and the faster the crawl got the more of them there were.

Now two maps: the newest run decides whether the row says "running", and the newest *finished* run supplies the numbers, so what is displayed is always something somebody counted.

Record the discovery channel that was inside the corpus already

08:25 pm AWST · 7a0acb8

Every marketplace listing carries a dealer_name. 842 distinct names across 7,778 listings, and 730 of them matched no source we held — the largest discovery pool available, sitting in a column we had been storing all along.

A trading name guesses a domain far better than a group name does: inventing domains from group names ran at 35% once and zero twice, and this ran at 19% across 89 probes. The misses cost nothing, because a domain that does not exist fails at DNS before a request is made.

Sixteen sources seeded today across WA, VIC, NSW and ACT. Four of them were reached only after stripping the "Pty Ltd" a marketplace carries in a trading name and a domain never does, and several only through the sitemap fallback added to probe-candidates this afternoon.

The argument for seeding a dealer whose cars we can already see: Zai Motors advertises 63 on Autotrader and 880 on its own site. Maddington Isuzu UTE, 69 against 377. Melville GWM, 48 against 173. The marketplace is a shop window, not the yard.

Expect the rate to fall below 15% from here — what is left is smaller dealers whose domains are less predictable.

A model year too thin to measure meant no age correction at all

07:44 pm AWST · 05530b1

Taj valued a Tesla and got $42,000 where the 2025 listings ask early-to-mid fifties. Reproduced: 2025 Model 3, 15,000 km, $42,959 against a 2025 median ask of $54,888.

The year shift only fired when *both* the subject's year and the comparable's had a measured price level, and levels needed four cars in a year. Model 3 2025 has three. So the subject's year had no level, no comparable was shifted, and the estimate fell onto whichever year had the most listings — 130 of the 143 comparables came from 2021-2023 and sat at their own years' prices. A 2023 Model 3 asking $39,888 was being treated as a 2025.

This is a regression from 0.3.0. The pooled kilometre slope used to smuggle in some of the age correction, because older cars carry more kilometres; splitting the terms removed that accident and then declined to replace it whenever the year was thin.

Levels now need three cars rather than four, and years still without one are filled from the median year-on-year step across the years that do — robust to a single odd year in a way a first-to-last slope is not — applied outward from the nearest measured year and no more than two years past the measured range. Beyond that the step is describing a market it has not seen, and the nearest real level is the better answer. Holes inside the range interpolate between their neighbours.

Correcting by an estimate is wrong by a little. Correcting by nothing was wrong by four model years.

Held out with the harness now importing the shipped function rather than its own copy, which is why this was not caught earlier: thin-year subjects -20.2% median error, dense-year subjects unchanged at $3,148 against $3,149. The Tesla reads $52,649 and the well-populated case is untouched — a 2023 Model 3 still returns $39,100 against a $39,888 median.

Note what the shutdown handler does not fix

06:39 pm AWST · 5abef75

Stopping the crawler to pick up the new rate settings tripped traversal. Each carried a dealer's own "this vehicle has sold" declaration, so the evidence was sound; what was missing was the precondition.

Delistings are written when a source finishes traversing and the run row is marked complete after that, so a kill landing between the two leaves real delistings attached to a run the handler then marks incomplete — which is exactly what the invariant is built to reject. The handler closes the lock; it cannot un-write the delistings.

The invariant was left alone. `complete` proves the index was fully readable, which is the thing that stops a site outage becoming a hundred fabricated sales, and weakening it to clear a failure I caused would be the wrong trade. The six were reverted by hand and will be re-detected on the next complete pass, since the cars really are sold.

The lesson is about when to stop a crawler rather than about the rule: stopping between sources is free, stopping mid-source is not. Recorded on the handler so the next person reaching for kill knows what it does and does not cover.

Crawl at the rate sites actually tolerate, and notice when they object

06:26 pm AWST · 980b9ca

sake of it — fast and properly, without costing data quality.

Three changes, in the order they matter.

**A 429 or 503 now slows the host down.** Nothing did before: the status was recorded as an error and the next request went out at exactly the same rate. That was survivable at eight seconds and would not be at two. backOffFor() doubles the host's gap for the rest of the run, honours Retry-After in either of its forms, caps at 60s and never decays back — a published Crawl-delay is a site's stated tolerance, 429 is its live one, and the live one wins. Requests are still not retried; a block is recorded, not fought. Only the pace of everything after it changes.

**The default gap drops from 8 seconds to 2.** It was undocumented, and with one request in flight per host and 24 sources at once it capped the whole crawl at about three pages a second — the ~8,600 an hour passes actually managed, and why a full pass of 678 sources took most of a day. Nothing had asked for eight seconds. A site's own Crawl-delay still wins wherever it is larger, which the fetcher enforces with a max() that no configuration can undercut.

**Jitter is proportional rather than flat.** A flat 0-1.5s was noise at 8s; at 1s it was most of the delay again, so a site asking for 1 second was quietly being given 1.75.

Both marketplaces move to 1s. CarsGuide publishes Crawl-delay 1, so that is its stated tolerance and it now gets exactly that. Autotrader publishes no rate directive at all, so its 1s is reasoned from the sibling site on the same platform — recorded in terms_notes as an inference rather than dressed up as compliance with something it never said.

Data quality is protected by the thing that makes speed safe rather than by staying slow: the rate is now a starting position that a site can correct.

Marketplace depth: what was actually limiting it, and a note that was wrong

06:18 pm AWST · 8222232

What was limiting coverage turned out not to be the rate or the budget. Measured over completed runs, Autotrader averaged 870 detail fetches against a 2,500 budget having found 3,841 listings, in 95 minutes — neither out of budget nor out of time, but out of URLs it had not already seen. Discovery depth was the ceiling. Index pages raised 400 to 1,500 and detail pages 2,500 to 8,000 on both marketplaces, which is affordable because the crawler runs 24 sources at once and a long run there occupies one slot.

Also corrects a note that was simply false. market:autotrader's terms_notes claimed "their robots.txt asks for Crawl-delay 1, so this stays twice as conservative as requested". Fetched today, Autotrader's robots.txt is Allow: / with disallows for deep paths, fuel-type filters and query strings, and no rate directive at all. That sentence is CarsGuide's — both sources carried byte-identical terms_notes, so it came across with the copy. The 2s we use is our own courtesy against no published rate, which is a different and weaker claim than the one the file was making. Both stay at 2s.

The arithmetic is in the doc rather than implied: 77,000 listings plus the index pages to find them is around 81,000 fetches against one host, four to six days of continuous crawling at the measured page rate. It accumulates over weeks. Making it arrive sooner means raising the rate, which is a decision about how this project treats a named company, not a technical one.

Stop fetching accessory catalogues as if they were cars

06:04 pm AWST · 9bcb625

First pass after 107 sources moved onto their own sitemaps: 367 "extractor returned null" errors, and the paths behind them were /pages 141, /new-vehicles 58, /models 41, /accessories 29, /special-offers 14, /click-to-buy 10, /blogs 7. Brighton Mazda found 301 URLs and extracted 80.

The sitemap fallback accepted any same-host URL with two segments whose last segment contained a digit — so /accessories/mazda2/ and /models/cx-5/ both qualified, because "mazda2" and "cx-5" contain one. Roughly 350 requests a pass to real dealers for pages that were never going to hold a car.

Two changes. A path list for pages that are about cars rather than cars, kept separate from NOT_A_LISTING because that one describes sections of a site and this describes pages that pass the shape test by accident. And the shape test now wants four consecutive digits rather than any digit, which is the rule discovery already applies to detail links — every real id shape in the corpus clears it, since stock numbers run six to eight digits and year-slugs start with a year.

Checked against eleven paths, five junk and six real, covering iMotor, Cox Radius, the /stock/details/ shape, easycars and the year-slug form.

This is the cost of the sweep working: a sitemap lists everything a site wants indexed, so the deeper the reach the more of the site's own furniture arrives with it.

Autotrader is in scope and has been for two days

06:01 pm AWST · 38a0a74

The disabled-sources table listed it under "carsales, Autotrader, Drive, Gumtree — WAF-blocked, out of scope". That was true of a 2026-08-10 survey which recorded a Peakhour 403 on everything, and stopped being true the next listings, the Disallow rules cover query strings and paths of nine or more segments while listing URLs are seven, and it shares CarsGuide's platform so the existing extractor reads it unchanged.

It has been enabled and crawling since — 5,176 live listings — while the document said it was out of scope. The durable record in source.robots_notes was right the whole time; the table was the stale copy.

Which is the same failure this section already documents for four dealer

Value a GLX against GLXs: trim gets its own term

05:44 pm AWST · b67d339

listing he could see was $35,000-$38,000. He was right, and the cause was that trim — worth more than anything else this model corrects for — had no term at all.

The comp set held 49 GLS, 46 Exceed and 9 GSR against 18 GLX. For 2021-23 those trims ask a median $38,990, $41,980 and $44,888 against the GLX's $35,990, so roughly three-quarters of the weight sat on dearer cars. Two reasons it could not right itself: badges were compared as exact strings, so "GLX (4WD) 5 Seat" counted as a different trim from "GLX", and a 1.5-against-0.8 weighting cannot outvote a set three-to-one against you.

It was bending the kilometre slope too, exactly as mixing model years used to. Pajero Sport: -$1,487 per 10,000 km pooled, -$999 within year, -$781 within year for GLX alone. Trim variation was being absorbed into the kilometre coefficient and then extrapolated across 90,000 km gaps.

Comparables are now restricted to the subject's trim wherever there are at least eight, and where there are not, the others stay and are moved onto the subject's trim by measured price level — the same correction that already handles model year. Badges are normalised first, so drivetrain and seat descriptors stop splitting a trim into strangers. The slope is centred within trim as well as year.

Held out over 31,939 cars in 20 trim-varying models: median error -31.7%, p90 -37%, median bias -$69. Per model: Mazda CX-5 -42%, Isuzu D-Max -39%, Nissan Navara -38%, Pajero Sport -33%. This is the largest single improvement the valuation has had.

The case that prompted it now reads $37,560 with 20 all-GLX comparables, and confidence rises from 0.84 to 0.90 because the comparables are finally comparable. A Pajero Sport Exceed asks $46,584 against the GLX's $37,560, which is the $9,000 the model previously could not see.

Normalisation is deliberately conservative: "Sport GLS QF" does not reduce to "GLS". Over-merging two trims that differ by ten thousand dollars is the more expensive mistake, so an unrecognised badge costs a comparable instead.

Also, both of Taj's display asks. The range now says "excluding government charges" under the number rather than only in the workings — a dealer holding this against the ads in front of them needs to know which money it is first. And confidence is stated, not implied by the width of a bar: a filled pill for high, outlined for low, with the score beside it. Yellow carries black, per BRAND.md.

Always value in excluding-charges money, and restate drive-away ads to match

05:16 pm AWST · a377c41

they are reading the prices from the drive away prices." Measured on 9,811 held-out cars advertised excluding charges, the range ran $353 high at the median and over-valued 53.2% of them. He was right.

A third of the corpus advertises drive-away and a sixth excluding government charges. valuation.ts computed on whichever basis dominated a comp set and down-weighted the other, converting nothing — so the answer depended on who happened to be advertising, and a car surrounded by drive-away ads was valued in drive-away money.

Now every comparable is restated excluding charges and the range always reports that basis. Drive-away ads lose their state's stamp duty; the slope and year levels are fitted on converted prices, so the whole model works in one currency rather than a blend.

The rates are legislated, not measured, and that was a decision. Comparing drive-away against excluding-charges listings of the same make, model, year, condition, kilometre band and state gives NSW +11.5%, VIC +10.1% — but QLD -1.2% and TAS -11.6%. A drive-away price cannot sit below the same car's excluding-charges price, so those pairs are not the same car; badge and trim vary inside a cohort and which cars a dealer advertises drive-away is not random. The corpus cannot calibrate this. Each schedule is read from its own revenue office, cited and dated in govt-charges.ts, with the arithmetic worked by hand in test-govt-charges.ts rather than asserted against the implementation.

SA and ACT return null and are left unconverted: RevenueSA blocks automated access so its schedule was only available from third-party sites, and the ACT's published table read "from 1 February 2027" when checked, which is not today's rate. This repository does not encode a tax rate from a blog or from next year.

Stamp duty only. Registration and CTP also sit inside a drive-away price and are not subtracted, because they cannot be computed honestly — registration turns on term and vehicle class, CTP on insurer and sometimes the driver, and a dealer may fold in three months or twelve. The residual shows up exactly where it should: bias falls from $353 to $175, and $175 is about what a third of comps each carrying $600 of un-subtracted on-roads would leave. It is disclosed on the page rather than closed with a guess.

Median error also improves 0.6%. Discarding drive-away comps altogether scores better again on both (bias $0, error -1.5%) and is not taken: it throws away a third of the comp set, which the well-populated models in this evaluation can afford and a thin cohort cannot, and its mean bias is -$594.

Read the dealer's address from the node that actually has it

04:37 pm AWST · 7c07c9b

suburb and postcode were 0% across all 58,991 live listings on the jsonld extractor, against 97% on dealer-studio. The field was being read correctly from the wrong node: schema.org puts the seller on the vehicle, and many of these sites put nothing there at all. melbournebmw's Car node has no seller whatsoever, while an AutoDealer node beside it gives South Melbourne, VIC, 3205 in full.

The vehicle's own seller still wins where it exists. The page's dealership is a fallback, and only when the page names exactly one business with an address — a group page listing several yards cannot say which one sold this car, so it contributes nothing rather than guessing.

Measured across ten sampled sources rather than assumed: five now yield a complete address (Bunbury 6230 WA, Southport 4215 QLD, Ringwood 3134 VIC, South Melbourne 3205 VIC, Chatswood 2067 NSW), two more yield a dealer name, three still yield nothing. So roughly half the platform, filling in as listings are re-observed rather than all at once.

Found while checking a 0% in the dealer-studio extractor test, which turned out to be a fixture artefact — that platform is at 97% in the live corpus. The fixture was wrong and the number beside it was real.

The sweep's other half: 273 delistings against 4

03:31 pm AWST · 11beac3

The sitemap work was done for coverage. Measured on the first pass to use it, its bigger effect may be the sold signal.

A twelve-card index cannot show that a car has left, because you only ever saw twelve. A sitemap lists the dealer's whole current stock, so a listing we hold and the site no longer advertises stands out. Previous pass: 388 sources, 4 delistings, 7 price changes. This pass: 73 sources, 273 delistings, 388 price changes.

The evidence guard held. 27 sources had delisting candidates rejected because the page still answered 200 — missing from the sitemap, not sold — Bayford alone with eight that would otherwise have been recorded as sales. A dealer's sitemap is a cache and it lags, and the confirmation fetch is the difference between evidence and inference.

Delisting is how this product infers a sale, so this is a better-founded sold signal and not only more rows.

Also restates the pace, because depth costs frequency: 169 pages per source against 27.5, about 32 sources an hour against 73, so a full pass is nearer 21 hours than 9 and the six-hourly cron will usually find one in flight. The reachable listings arrive over days. That sharpens the hosting question rather than softening it.

Bump the model version, because the model changed

01:01 pm AWST · f7aac51

MODEL_VERSION is stored on every valuation row and is the only thing distinguishing one generation of the arithmetic from the next. The km/year split shipped an hour ago without touching it, so valuations computed this morning under the pooled slope and this afternoon under the split terms would both be filed as spotlot-val-0.2.0 and be indistinguishable to anyone reading the table later.

Nobody was served a stale number — model_version only dedupes the stored record, and the range itself is always computed fresh — but the stored history would have been quietly wrong about what produced it.

0.3.0 carries the reason with it, so the next reader does not have to find this commit to know what changed.

Also fixes the API docs page, which advertised spotlot-val-0.1.0 — two generations behind what the endpoint actually returns.

A killed crawler should close its own run rows

12:56 pm AWST · bc4569b

I stopped the 06:43 pass deliberately at 12:02 so the next one would pick up the new sitemap configs. The 12:43 pass then refused to start:

another crawl is in flight (dealer:accars since 13/08/2026, 12:02:02 pm)

It was not in flight. The process was gone; its 22 open crawl_run rows — one per source in the concurrency window — were not. anotherCrawlIsRunning() treats any unfinished row started within STALE_RUN_HOURS as live, and closeAbandonedRuns() only reaps rows older than that same window, so nothing could clear them for two hours. The guard was working correctly on a lie, and it cost the exact pass the stop was performed for.

crawl.ts now tracks the runs it opens and closes them on SIGINT/SIGTERM, marked incomplete — a killed traversal saw part of a site, and a partial traversal must never let a delisting be inferred, which is the rule the budget-exhausted path already follows. Rows closed normally are deregistered so the handler has nothing to do for them.

This does not replace closeAbandonedRuns(): a laptop that sleeps or a SIGKILL still leaves rows behind, and the two-hour reaper is the backstop for that. It removes the case where the crawler was asked politely to stop and left a lock behind it anyway.

Correct age and kilometres with separate terms, not one doing both

12:37 pm AWST · cd0f3da

Taj asked what the adjusted price represents and whether it skews valuations. It does, and the fix is not the one I first proposed.

The valuation shifts each comparable to the subject's odometer using a Theil-Sen slope fitted across the whole comp set. That set spans model years, and because older cars carry more kilometres the slope absorbed age depreciation too: pooled it is 1.6x to 3.7x steeper than the same-year figure on every high-volume model (Ranger -2,479 against -862 per 10,000 km; Sportage -2,657 against -716). Nothing else adjusted price for age — year only reduced a comparable's weight — so one coefficient was carrying both effects and over-shifted any subject whose odometer was unusual for its year.

My first suggestion was to fit the slope within years and stop there. Held out over 46,576 cars in 40 models, that is 0.5% WORSE: stripping age out of the slope while adding no age term leaves age uncorrected. Recorded because it was the obvious fix and it was wrong.

What works is both terms. Kilometres measured within years, plus each model year's median ask at a reference 80,000 km, so a comparable is moved in years and in kilometres separately. Held out on identical comp sets and weights:

slope median err p90 err mean err none $4,102 $13,993 $6,143 pooled $3,241 $11,198 $4,843 <- what shipped within $3,257 $11,541 $4,944 km+year $2,804 $10,534 $4,410 <- this

13.5% better median error, and better on all 12 of the highest-volume models individually. The cap moves 25% to 35% because the shift now carries two effects; isolated, the year term alone is worth 11.5% and the wider cap alone is 3.7% worse, so the extra room only helps once there is something legitimate to put in it.

What it does to the number on screen, which is a separate question from p10 -8.3% and p90 +8.3%, and 13.5% of cars move by more than 10%. Those are the cars whose odometer sits far from their comp set, which is exactly where the single coefficient was wrong.

scripts/eval-km-slope.ts is the harness, kept so the claim can be rerun rather than believed. It predicts advertised prices, which is what this tool estimates and all Australia publishes — it does not prove the estimate is closer to what a car sells for.

Entry 13: orange has now held three readings

12:05 pm AWST · 860531d

The history table added this morning has its third row, and the winner has not fit growing from 88,407 cars to 94,845.

That is more than gold managed — it held for two readings and then became orange, which is what prompted the entry. Three in a row across an 8,000 listing increase is not proof the effect is real, but it is the first time this insight's identity has been observed holding still rather than recalled as having done so.

Rover is a car; and let the sweep see badges too

12:01 pm AWST · 31fd834

Two things arrived in the same review batch and needed opposite treatment.

A Rover 75 Cdti came in from CarsGuide. Rover is a real British marque and was missing from the canonical list, so it was being held for a human and would have been dropped — the opposite failure from the caravans beside it. Added, with a note that a bare "Rover" means this marque: feeds that mean Land Rover say landrover, land-rover or range rover, all of which are already aliased.

The four caravans — Empire Luxor, Elite Eildon, Vivid Caravans Nomad, Snowy River SRC21S, every one badged CARAVAN — were caught at ingest by this morning's badge rule but not by recanonicalise, which only tested make and model. So rows already stored under the old rule sat in the queue forever waiting for a human to decide something the rules already decide. The sweep now applies the same badge test as ingest, which is the point of having one vocabulary rather than two.

Queue is clear and all ten integrity checks pass.

Entry 16: the same bug four times in one day

11:30 am AWST · b9ea94b

const { data } = await client.from(...) — the error discarded — produced four confident wrong answers on 2026-08-13: probe-sitemaps reporting "no sources matched" for 119 sources whose id=in.(...) URL was too long; a throwaway audit saying "pending aliases: 0" when five were pending and the column names were wrong; check-integrity printing "-1 row(s) ... against -2" when the data was perfect and the RPC had timed out; probe-candidates calling a dealer unusable for HTTP 404 when it publishes a sitemap nobody asked for.

supabase-js resolves either way and returns { data, error }. Take only data and a failed query becomes indistinguishable from an empty result — and since empty results are normal, every caller already has a plausible branch waiting for it. The failure does not look like a failure; it looks like a fact.

This codebase is unusually exposed to it because absence is meaningful model and it makes the empty branch the well-trodden one.

Records what is already defended — the ingest path drops errors too, but unique indexes on vehicle.vin and vehicle.fingerprint turn a swallowed error into a constraint violation rather than a duplicate vehicle, which is luck rather than design and is why the crawler was not rewritten in a hurry today.

52 call sites still destructure only data. Sweeping them is a day's careful work across the crawler's core, and doing it fast to close an entry would be the wrong trade on a live corpus. The two producing wrong answers were fixed.

An integrity check reported corruption it never observed

11:01 am AWST · f4e122c

check_fields_empty_flag() counts observations whose flag disagrees with their payload across 393,000 rows. Through PostgREST it now exceeds the API role's statement timeout; run directly it answers in seconds and says the data is perfect — 183,867 flagged as content, 183,867 actually content, 0 mislabelled.

check-integrity.ts destructured only `data`, so the timeout arrived as null and the sentinel defaults were printed as counts:

FAIL every observation's fields_empty matches its payload -1 row(s) flagged empty while holding content; -1 flagged as content against -2 that are

Negative rows are impossible, which is the only reason this was obvious rather than alarming. The failure mode matters more than the timeout: a check that cannot run must say so. Reporting a failure it never observed sends somebody hunting corruption that is not there, and teaches the next reader that a red line in the integrity suite is noise — which is the one thing that suite cannot afford.

0080 gives the function a 60-second ceiling of its own, following 0073 and 0079. Sixty rather than fifteen minutes because this one runs inside a request, and a check that takes a minute is already saying something.

Third statement-timeout failure this week, and the third time today that a discarded error turned into a confident wrong answer.

The daily rollup had been failing silently for two nights

10:46 am AWST · 9ab664a

pg_cron's daily-rollup job died at exactly 00:02:00 on 2026-08-11 and 2026-08-12 — "canceling statement due to statement timeout". write_daily_rollup joins listing_event to listing_current and scans it again for the day's delistings, and at 137,000 live listings that outgrew the cron role's 2-minute default. Measured after the fix: 188 seconds.

Migration 0073 gave depth-boards a 15-minute ceiling for exactly this reason and left this job on the default. 0079 does the same for it. Two jobs, one shape of problem, fixed six days apart because the first fix was scoped to the job that happened to be failing rather than to the class of job.

Nothing alerted for two days, because a cron job that fails silently looks exactly like one with nothing to do. It was caught by 0077's stale-board check noticing the tables were 36.3 hours old — which is the argument for that check existing.

2026-08-12 has no rollup and is not being backfilled. The function takes a date, which makes a backfill look safe; only cuts, rises and confirmed_exits are filtered by it, while live counts, vehicle counts and every ask quantile read current state. Writing today's market under yesterday's date would produce a row that looks perfect and is a lie. The gap is recorded in DATA-QUALITY entry 15 instead.

Annotate the survey's "browser UA" so it is not read as licence

10:05 am AWST · 8c1ded9

docs/research/dealer-site-directory.md records its method as fetching each site with a "standard browser UA". That was a one-off manual pass on 2026-08-10, before the crawler existed, and it contradicts a rule at the top of this repository.

Verified rather than assumed that it describes nothing current: fetcher.ts sends SpotlotBot/0.1 and there is no browser user-agent string anywhere in src/ or scripts/.

The paragraph is left as written — it is a record of what was done, and rewriting it would make the tree tidier and the record false. The note says plainly that it is history, not practice.

Both sweep batches done: 15,542 listings reachable, 105 sources

09:56 am AWST · ccba77e

Batch 2, run against the 82 sources that were not iMotor-shaped but sat under 25 listings or carried an unmatched URL shape: 68 have a usable sitemap, 30 gain more than 20 listings, +5,177. Footscray GWM Haval held 8 against 530, MMG Auto 11 against 504, the three Rex Gorell franchises 332, 319 and 264.

With batch 1 that is 105 sources now discovering through their own sitemap and 15,542 listings reachable that were not yesterday.

Reachable, not held — and fetching them is the constraint this document exists to measure. At 200 detail pages per source per pass and four passes a day, with the rotation taking unseen listings first and stalest second, the backlog clears over days. Nothing here raises maxDetailPages: a pass cannot spend more than its budget, so the cap is not what is binding.

Defer the freshness clock's first sync out of the effect body

09:30 am AWST · 9b0f44d

eslint flags setNow(Date.now()) called synchronously inside useEffect, and it is right: that schedules a second render inside the first.

The sync itself has to happen. The server renders `now` as computedAt, so a page served from the CDN at age 40s would display "0s ago" until the first tick — which is the opposite of what a staleness indicator is for. Moved to a setTimeout(0) alongside the existing interval, with both cleaned up.

Also drops exactCount from queries.ts. It was orphaned when the eleven separate COUNT(*) round trips were replaced by the coverage_counts RPC, and has been dead since.

tracked probe-tmp.ts from an earlier investigation, deliberately left alone.

Reject the all-nines odometer, keep the zero one

09:02 am AWST · 619d9c3

A 2010 Camry is advertised at 9,999,999 km — typed by whoever built the listing to mean "not entered". Stored as a number it is a reading of ten million kilometres, and it feeds medians, the price-by-age-and-kilometres fit behind every insight, and the comparables behind a valuation.

The ceiling is 1,000,000 rather than something tighter because the corpus holds real high-kilometre commercial vehicles and their distribution is continuous: 799,686, then 704,323, then 636,700, all the way down. Only the sentinel sits outside it, in 13 observations of one listing.

Null rather than clamped — we do not know this car's odometer, and a blank field stays blank.

Zero is explicitly kept. The first version of this rejected it, which would have blanked the odometer on 3,545 live listings that read 0 km because they are new cars, where zero is the reading and not a missing value.

Correct the correction: /stock/details/ does buy listings

08:59 am AWST · e3a763b

This section recorded that shape as worth zero, on three measurements that were all accurate: Melbourne BMW 557 held against a sitemap of 558, North Shore BMW 314 against 315, Waverley BMW 417 against 416.

Every one of those three was already fully discovered by walking its index pages, so the sample could not have returned anything but zero. Run across under-covered sources instead, BMW Sydney goes 27 to 252 and Castle Hill BMW 30 to 175.

What decides the value is not the URL shape but whether that source's index traversal was already reaching its stock — which is a property of the source, not of the pattern.

Third time today I have generalised a measurement past the sample that supports it, so the lesson is now stated as the general one: measuring "does this help?" on subjects that need no help returns no, every time.

The iMotor sweep finished: +10,365 listings reachable

08:40 am AWST · e7330cf

119 sources probed, 115 publish a usable vehicle sitemap, 75 gain more than 20 listings, total upside 10,365. All 75 now discover through their sitemap with the HTML stock page kept as a secondary index, so a truncated pass still refreshes prices.

Trivett held 33 against 1,131 in its own sitemap. Llewellyn MG 13 against 507. Jarvis Ford 12 against 387.

The sources already well covered gained nothing, which is the result that makes the rest credible: AMR Used Cars 14 held against a sitemap of 13, Llewellyn Motors 645 against 504. A sitemap lagging what we hold is normal — it is a cache of the site's own index, not a second opinion.

Replaces the partial figure of 38 sources this section carried while the run was still going.

"Unknown" is a placeholder, not a marque

08:39 am AWST · 28004a3

Four Penrith sites each publish /cars/used-grey-2026-unknown-unknown-4623 — make "Unknown", model "Unknown", no price, one stock id repeated across the group. Their CMS emitted it. It is a real page and not a car anyone can value, and it was sitting in the review queue on all four.

Excluded by rule rather than by hand. The review queue is supposed to mean "a human still has to decide this", and this is now decided; and check-integrity requires every out-of-scope vehicle to be one the rules exclude, so a hand-set flag would fail the invariant built to catch exactly that kind of edit.

The two caravans in the same batch need no new rule — they are already caught by the badge check added this morning. They reappeared only because the running crawl loaded its code at 06:43, before that change existed, and will stop arriving once the next pass starts.

Ask the site for its sitemap before declaring it unusable

08:38 am AWST · a52acdb

probe-candidates guesses inventory paths and, when every guess 404s, reports "inventory HTTP 404" and moves on. That verdict has already been wrong in an expensive way: dealer:darwingwm and dealer:riverinavolkswagen were both withdrawn on exactly that note, and both publish a stock sitemap naming 164 and 96 vehicles, linked from their own robots.txt. A guessed path answering 404 says nothing about whether a dealer publishes its inventory.

It now falls back to the site's own sitemap, requires at least 10 vehicles in it, and then opens one through the existing confirmVehicle check so the verdict still names the extractor and a real car with a price — counting links proves a sitemap exists, not that anything can be read out of it.

Of the 20 candidates probed this morning, exactly one is rescued by this: kerryholden.com.au, 42 vehicles, previously discarded as HTTP 404.

The patterns and the sitemap walk now live in src/lib/crawl/sitemap-scan.ts instead of being copied into a second script. They have been wrong once in a way that cost 97 sources, and two copies drifting apart is how that happens twice. probe-sitemaps re-verified against Ipswich Chery after the extraction: 13 held, 129 in sitemap, 3 fetches — identical to before.

Kerry Holden is deliberately NOT seeded despite passing. All 42 of its stock ids are already inside Darwin GWM's 164 — same group, same Darwin stock on a second domain — so seeding it would buy 42 duplicate listings and spend the fetches twice. That is the containment problem in docs/COVERAGE-CEILING.md, and finding a usable site is not the same as finding a useful one.

Name the row instead of reaching for the last one pushed

08:10 am AWST · ff11f78

applyOne was being handed rows[rows.length - 1], which is the row just pushed only for as long as nobody adds a second push or reorders the loop. Bind it to a variable and pass that.

DEMO: date the duplicate measurement so it stops half-reconciling

08:09 am AWST · 3804bcb

The headline says 129,286 live and the paragraph below said 132,730 rows minus 3,540 duplicates, which is 129,190. Both are right and they were taken twenty minutes apart, so a reader checking the arithmetic finds a 96-listing hole and no way to tell which number to trust.

Stamped with the time it was measured, and says plainly that both move with every crawl while the 2.7% is the durable part.

Measure the model pages before they become the next /market

08:03 am AWST · 4662dd0

/model/toyota/hilux is 5.0s cold, 0.56-1.16s warm. model_ticker alone is 1.95s and 74,646 buffers for a single model.

The cost is not the make/model lookup — vehicle_make_model_idx covers that. It is listing_current_lean resolving each listing's latest observation at roughly 37 buffers per listing, which is exactly what migration 0066 recorded about the unlean view: reading it is O(n x per-row work). The uncomfortable part is that n is how many of that model the corpus holds, so a popular model gets slower every time coverage improves.

Not fixed here. cached() absorbs it for everyone after the first visitor, so it is a cold-path cost rather than a live failure, and the real fix is a denormalised current-state column — a design change that overlaps work already in flight on that view. Recorded now so the measurement exists before it turns into a page that 500s, which is how /market announced the same problem.

DEMO: state freshness as a share, so it stops contradicting the headline

08:01 am AWST · 8919cc0

The paragraph said "of 132,903 live listings" four lines under a headline saying 129,286. Both are right — one counts rows, the other distinct pages — and a reader totalling them would reasonably conclude one is wrong.

Stated as 88% instead, with the row counts in parentheses and a pointer to the paragraph that explains the difference.

DEMO: the freshness number, which was measurable and unstated

08:00 am AWST · 7ff1f86

Of 132,903 live listings, 116,449 were re-observed in the last 24 hours and every remaining one within three days. Nothing older exists in the corpus — not mostly, zero.

This is the whole thesis of the product stated as a number, and it was nowhere in the demo script. "Most tools photograph the market once; Spotlot watches it" is a claim; "every live listing was seen again within three days" is the evidence for it, and it costs one query to check in front of somebody.

Included the query so it can be run live rather than believed.

Refresh figures: 129,286 live across 677 sources

07:51 am AWST · ccd5677

Distinct-URL counts throughout, matching what the site now publishes. Sources 677 rather than 672 — the five recovered this morning: darwinmazda, darwingwm, riverinavolkswagen, carlinandgazzardvolkswagen, and northlakesgwm, which was not in the corpus at all.

Makes 85, down from 87, as the taxonomy sweep took Bad Boy Mowers, two caravans and a Western Star out of scope.

Count cars, not rows, everywhere the corpus size is published

07:48 am AWST · 9b7ea4e

coverage_counts() feeds the number on the home page, /market, /value and /api, and it was counting listing rows. 132,730 live rows resolve to 129,190 distinct URLs; the other 3,540 are the identical page under a second source id, so every one of those surfaces read 2.7% high.

Distinct URL is the right key rather than the convenient one. Two rows sharing a URL are the same page on the same dealer's site, whatever id discovery filed them under. Two different pages advertising one car — a marque site and its group site — keep their own URLs and are still counted separately, which is correct at this level: the corpus does hold two listings, and valuations dedupe by vehicle where it matters.

This fixes the arithmetic, not the duplication. The redundant source ids still exist and are still fetched twice; retiring them without inventing delistings is entry 14.

Live figure after the change: 129,234.

Entry 14: 3,540 live listings are the same page counted twice

07:45 am AWST · 39060fe

132,730 live rows resolve to 129,190 distinct URLs. The gap is not one car listed on two sites — it is the identical URL stored under two source ids, because a single dealership was seeded more than once: wynnumgwm and bartonswynnumgwm hold the same 364, capalabagwm and bartonscapalabagwm and bartonsgwmhaval the same 250, brightonkgm and brightonssangyong the same 116. The reasons are mundane. GWM dealers renamed from "GWM Haval", KGM is what SsangYong is called now, a group prefix appears on some entries and not others, and ftgchery is cheryferntreegully abbreviated.

COVERAGE-CEILING measures containment and concluded duplication costs fetches rather than correctness, since valuations dedupe comparables by VIN. That holds for a group site restating a marque site's cars under its own URLs. It does not hold here: an identical URL under two ids inflates the count, and the count is published on the home page.

Not fixed yet, and the reason is in the entry. Disabling one id of each pair stops the duplicate fetches but leaves the duplicate rows live, and a delisting has to carry evidence — a config change is not evidence. That needs a migration that retires the losing id's listings, written carefully.

What was safe to do immediately is stop publishing the inflated number, so README and DEMO now quote distinct URLs and say so. DEMO also notes that the pages themselves still count rows and therefore read 2.7% high until the merge lands, which is the sort of thing a demo should say out loud rather than hope nobody totals.

Re-enable four sources disabled by a filter, not by a site

07:40 am AWST · 83d60d0

dealer:darwinmazda sat on the disabled table reading "stock sitemap lists category pages, not vehicles". Its sitemap lists 106 vehicle detail URLs and 3 category pages. The check that disabled it read the file through the sitemap sweep's VEHICLE_PATH filter, which cannot match an iMotor detail URL — so the three it could see were the whole picture it got. The note was a true description of a filtered view and a false description of the site.

Re-checked and re-enabled, each with a detail page re-probed for window.pageData, stockItem and vehicle JSON-LD: darwinmazda (106), darwingwm (164), riverinavolkswagen (96), carlinandgazzardvolkswagen (35).

The two "every inventory path answers 404" notes were not wrong, just incomplete — every guessable path does 404, and the sitemap named in the site's own robots.txt lists the stock anyway.

dealer:villagegwmhaval stays disabled with its reason corrected. It is not "new models only": its sitemap lists 188 vehicles, every one on northlakesgwm.com.au. The dealership was rebranded, so that source was reading a site that had moved. Added as dealer:northlakesgwm — a real dealer nobody was crawling — rather than re-enabling a row whose id names a different business.

Sibling LDV and RAM sites re-checked the same day and left disabled; their stock sitemaps genuinely return nothing.

The lesson for the table: a withdrawal note records what a check concluded, not what is true, and outlives the tool that produced it. Anything disabled for "no stock found" — rather than a 403, a WAF, or a robots disallow — deserves re-running whenever discovery changes.

Record the sweep's blind spot and what it was actually worth

07:28 am AWST · 3207441

97 sources sat at ~13 listings each because the sitemap sweep reported "sitemap 0" for them, which reads exactly like a dealer with no sitemap. Thirteen was not arbitrary: iMotor index pages server-render 12 Algolia hits and load the rest client-side, so walking the page sees twelve cars however many the yard holds. Ipswich Chery renders 12, holds 13, and reports nbHits 129 in the JSON its own page ships.

Partial run, 38 of 119 sources: 24 gain, +2,776 listings.

Also records the audit that followed, because the method matters more than the recognise? That table is what turned up cox-radius at 7% — and a platform at 7% is more dangerous than one at 0%, because the accidental matches make it look like something that works.

And the mistake it exposed. "URLs the sweep cannot match" is not "listings we are missing". The /stock/details/ shape looked like the biggest of the three at 3,287 listings across 35 sources; measured upside was zero, because those dealers were already fully discovered by walking their index pages. The equivalence was true for iMotor and I carried it across two more shapes without re-checking it.

Write each sitemap as it is found, not all of them at the end

07:27 am AWST · 41ac606

A 119-source --apply run died at source 34 and wrote nothing. Every update was still queued behind 85 sources of fetching, so an hour of polite, rate-limited requests produced no durable change and the only evidence it had happened was a log file.

The batching bought nothing to begin with — the fetches are the expensive part and each update is a single row. Now an interrupted run keeps what it has earned and re-running costs only the sources it did not reach.

The same run also has to be started properly. `nohup … &` from a tool call stays in the caller's process group and dies with it; this one is double-forked into its own session with PPID 1.

The /stock/details/ pattern buys nothing, and that is the finding

07:22 am AWST · 27d9dfb

Commit 1b6eb1f said "35 more sources, 3,287 listings", which reads as upside. Measured upside: zero. Melbourne BMW holds 557 against a sitemap of 558, North Shore BMW 314 against 315, Waverley BMW 417 against 416. Those sites were already fully discovered by walking their index pages.

The mistake is worth naming because I made it twice in an hour. "Listings whose URLs the sweep cannot match" is not "listings we are missing". For the iMotor sources the two coincided — 13 held against a sitemap of 129, against an nbHits of 129, on source after source — and I carried that equivalence across to two more URL shapes without re-checking it. The shape analysis finds where the sweep is blind; only a probe says whether anything is behind the blindness.

The pattern stays. A source reporting "sitemap 0" is indistinguishable from a source with no sitemap, and that ambiguity is precisely what hid the iMotor dealers; making the report truthful for 35 sources is worth a regex even when it moves no listings. --apply writes nothing for them regardless, since it requires the sitemap to beat holdings by more than 20.

Net of today's three patterns: iMotor is the real one, Cox Radius is +289, and this is 0.

Correct the Cox Radius claim: +289, not thousands

07:18 am AWST · 75abc00

The comment added in d137b75 led with "Patterson Cheney holds 423 live listings; its /all-stock/sitemap lists 1,931", which reads as though the sweep could not see that dealer. It could. Patterson Cheney publishes a second sitemap, /used-vehicles/, whose URLs are shaped /used-vehicles/<slug> and match VEHICLE_PATH — the sweep has been reporting 3,668 for the site since yesterday's run, as docs/COVERAGE-CEILING.md says in the section I had just read.

Measured properly, on the four sources whose only shape is /view/<title>/<id>: BDK Automotive 126 held against 242, Select Autos SA 50 against 98, Toowoomba GMSV 19 against 144, Xtreme Car Sales 96 against 96. That is +289.

Xtreme landing exactly on its 96 is worth keeping: a pattern that invented matches would have overshot there, so parity is evidence the other three numbers are real.

The pattern stays — 289 listings is worth a regex — but the number in the repository should be the one that was measured, not the one that made the finding sound bigger.

Add the /stock/details/ shape: 35 more sources, 3,287 listings

07:10 am AWST · 1b6eb1f

Third shape found by the same audit. Most of the BMW franchise sites publish detail pages as /stock/details/OAG-AD-26173410/2026-bmw-x3-used, and one wholesaler uses /wholesale-stock/details/… . VEHICLE_PATH wants a single segment after "stock" and these carry three; the year-slug alternative needs a hyphen followed by a digit and these end in letters.

Matched on the /details/ segment plus two following segments rather than the literal "OAG-AD-", so a site that renumbers its ads keeps working.

Checked the remainder rather than assuming it was all upside: of the 8,096 listings still unmatched, 7,650 are Autotrader and CarsGuide, which are marketplaces crawled by facet and have no business in a sitemap sweep. What is left is a long tail of one-off shapes across 22 small dealers — 96 listings at the largest — and is not worth a pattern each.

Assert the boards are fresh, because a stale one fails silently

07:08 am AWST · c98c7d1

Every heavy aggregate now reads a precomputed board, and every page degrades to an empty panel rather than a 500. Together those two make the worst kind of failure: if a refresh stops running nothing throws, the boards simply age, and /market reports "No live listings yet" over a corpus of 131,000 cars until a human happens to look. That is exactly defect 8's shape, an absent number with a comfortable explanation, and it already happened once today.

check_stale_boards() covers all seventeen boards and rollups, table driven so a new board is one row rather than new code. It allows two missed cycles before complaining: 12 hours for the 6-hourly depth and market boards, 36 for the daily rollups. Four boards whose own thresholds can legitimately exclude every row, like a state needing 300 price pairs, are checked for staleness but not for emptiness, because empty is a real answer there.

health.ts prints boards=ok on the quiet line, so a passing run states the aggregates are current rather than merely not having heard otherwise, and raises one ALERT per problem board with its age and limit.

Verified by making it fail rather than by trusting it: a board backdated 20 hours produced the ALERT and exit 1 that fails the workflow, and the timestamp was restored to the exact value it held before.

Teach the sweep Cox Radius detail pages too

07:06 am AWST · d137b75

Having found the iMotor blind spot by accident, I asked the question properly: of every platform, what share of its stored listing URLs would this file's patterns recognise? dealer-studio 100%, nextjs-embedded 98%, jsonld 91%, easycars 86% — and cox-radius 7%.

The 7% is the tell. Those were titles that happened to contain a hyphen followed by a digit, which `\d{4}-[a-z0-9-]+-\d` matches by accident: a "2025-BYD-Sealion-8-U8055400" scores, a "2021-Mercedes-Benz-CLA-Class" does not. A platform at a clean 0% invites investigation; one limping along at 7% looks like it works.

Patterson Cheney holds 423 live listings. Its /all-stock/sitemap, linked from robots.txt the whole time, lists 1,931 detail URLs. The sweep was already following that file — its child filter matches "stock" — and then discarded every URL in it.

CarsGuide also scores 0% and is deliberately left alone: it is a marketplace crawled through facet pages, not a dealer site with a stock sitemap.

Teach the sitemap sweep to see iMotor detail pages

07:04 am AWST · 422e090

97 sources sat at roughly 13 listings each while their sitemaps listed the whole yard. The sweep had been reporting "sitemap 0" for every one of them, which is indistinguishable from a site that has no sitemap, so they were skipped and nobody looked again.

Two filters were wrong in opposite directions. VEHICLE_PATH expects the vehicle-ish segment followed by one more segment and the end of the path, so /new-cars/for-sale/chery/tiggo-7/2025/urban/t32-my26/3841785/ — seven segments of make, model, year, badge, series — matched nothing. BROCHURE_PATH then excludes everything under /new-cars/, which is the prefix iMotor uses for genuine new stock, so a widened VEHICLE_PATH alone would have discarded the new-car half as brochures.

IMOTOR_DETAIL requires both a /for-sale/ segment and a numeric stock id at the end, which the model brochures BROCHURE_PATH exists to catch have neither of, and is tested before the brochure rule. /new-cars/tiggo-7/ is still a brochure.

Measured, not assumed: Ipswich Chery holds 13, its sitemap lists 129, and its own search index reports nbHits 129. Frankston Kia 11 against 222, Dubbo Hyundai 9 against 57. First six sources of the full run: three gained 107, 261 and 366; the three already well covered gained nothing, which is the result that makes the other three believable.

Also fixes the same swallowed-error pattern in the source query. 119 ids built an id=in.(…) query string long enough to be rejected, and only `data` was destructured, so the rejection arrived as an empty list and the script printed "no sources matched" — which reads exactly like ids that matched nothing. One id worked, so the obvious conclusion was that the ids were wrong. The query is now chunked, the error is thrown, and ids that genuinely do not resolve are named rather than counted.

Correct the source count: 672 dealers, not 675 of 682

07:03 am AWST · 31b7c15

The refresh an hour ago said "675 dealer sources holding stock of 682 enabled". Both numbers were counts of the wrong thing. The 682 is every enabled row in `source`, which includes the eight government datasets and the two marketplaces; calling those dealer sources inflates the figure with things that are not dealers. Broken out by kind: 672 dealer_site, 2 marketplace, 8 gov_dataset.

The right claim is stronger anyway — all 672 dealer sources currently hold stock, so the qualifier "holding stock" was carrying no weight.

Catch a make the feed split, and a caravan carrying no body type

07:02 am AWST · 80d5c16

Five aliases sat in the review queue and none needed a human.

"western star" has been excluded since trucks were, and a Western Star still matched nothing. Same failure as the "Avida," punctuation bug — the list was right and the key never reached it. isNonCarMake now optionally takes the model and tests the joined string against the same list, so it can add a match but never invent one.

A caravan arrived as make "Empire", model "Luxor", badge "CARAVAN" and no body type at all, so the body-type list was consulted against an empty field. Badges are now checked against their own small vocabulary rather than reusing NON_CAR_BODY_TYPES, because that list holds "cruiser" for motorcycles and a feed putting "Cruiser" in a LandCruiser's badge would have dropped a real car. Only words that are never a trim level are in it.

Bad Boy Mowers are mowers.

The model is threaded through recanonicalise and check-integrity too, so the sweep and the invariant agree with ingest instead of disagreeing quietly — an out-of-scope row the integrity check cannot re-derive is exactly the orphan that check exists to catch. Queue is empty: 4 vehicles swept, 3 aliases marked.

Keep what each insight said, so its stability can be measured

06:46 am AWST · fd6d87e

Entry 13 recorded an insight that changed its winner between runs — gold, gold, then orange — and deferred the fix as a judgement about what to show. This is not that fix. It is the prerequisite, which turned out to be missing: nothing in the database could answer "has this insight named the same winner twice?"

compute-insights deletes every market_insight row and re-inserts. That is right for the live surface, since an insight that stopped clearing the bar must not linger because it once did, and it destroys the claim on every run. market_insight_run recorded that 1 of 4 passed but never which, so the only record of gold becoming orange was a human reading the logs at each step.

market_insight_history is append-only, consistent with observations: an insight's history survives its own retraction. Verified by running compute-insights for real rather than seeding a test row — two readings so far, orange both times, which is one repeat and not yet stability.

Also records why the migration file carries no header comment. It was applied through the API with the prose trimmed, and sync-migrations regenerates files byte-exact from what the database stored, so the header could not survive. Putting it back by rewriting the recorded SQL would make the tree prettier and the audit trail false. The reasoning sits in entry 13, which the table's own comment points at, and OPERATIONS now warns the next person to apply the whole file.

Explain the freshness clock reading older than its own window

06:46 am AWST · 75a7643

The panels revalidate every 60 seconds and a visitor can still land on one saying "9m ago", which reads as a broken timer and was reported as one.

ISR regenerates on request, not on a timer: when the window expires nothing happens, and the next request is served the stale copy while the rebuild runs behind it. With no traffic there is no next request, so the page ages until someone arrives — and that person is always the one who sees the old number.

Measured on production rather than asserted. Four requests, no other change: age 365 HIT, then 15, 19, 22, 25. The first request paid for the rebuild and the second collected it, which is exactly what the 30-second AutoRefresh tick does for a tab left open.

Recorded so the next person to notice it can stop in one paragraph instead of going looking for the bug.

Refresh the corpus figures, including the two that went down

06:46 am AWST · 164e9af

The docs claimed 77,300 listings across 635 sources. The database says 130,991 live of 132,657 seen, across 675 sources holding stock of 682 enabled, with 357,912 observations, 3,096 price changes and 1,666 delistings. Being 42% adrift understated the product, but a number the database disagrees with is wrong in the direction that matters regardless of which way it leans.

Two figures fell and both are recorded rather than quietly dropped. Makes went 112 to 87 because the taxonomy sweep reclassified caravan, camper and truck brands that were never cars; a make count that only ever rises is one nobody is checking. Field coverage slipped a point or two across state, fuel, transmission and body because the sources added are small franchise sites publishing less per listing than the large yards already in the corpus.

DEMO also now separates "delisted" from "sold". A listing leaving a dealer's site is evidence the car is gone, not proof it was sold — 172 of them came back under a new URL. The pages say sold; the doc says delisted and states the gap, which is the honest width of the claim.

Every figure carries the timestamp it was taken at, so the next reader can see how old it is instead of trusting it.

Entry 13: an insight that kept changing its mind

06:46 am AWST · 4859ac9

Over one night, as the corpus went from 50,000 listings to 122,000, the colour-premium insight said gold at 10% (p=0.0478), then gold at 7% (p=0.0244), then orange at 7% (p<0.001). The regional insight went QLD, then NSW, then failed to clear the bar at all.

The p-values improved while the subject changed, which is the part worth naming. adjustForSelection corrects for choosing an extreme from many candidates — it is the right correction for *whether* the extreme is real and says nothing about *which* one. The argmax of thirteen noisy colour groups moves as data arrives; gold to orange is a different claim, not a sharper one.

It matters more than a wrong number because "gold cars command a premium" is a line a dealer repeats, and a strong p-value on the winner makes it read more trustworthy rather than less.

Deliberately recorded without a code change. The fix is a judgement about what to show — require the same winner across several computations, report the distribution instead of the extreme, or keep only pre-specified comparisons — and the most honest option is the least interesting to read, which is the actual tension. Until it is decided, the caveat stands: an insight naming one winner from many candidates is evidence of an effect size, not a fact about that winner.

"Avida," did not match "avida", and seventeen caravan brands

06:46 am AWST · 3da6600

The review queue filled to 22 overnight, almost all from one dealer that sells caravans beside its cars. Each arrived with a body type already excluded — caravan, pop top, hybrid — so no vehicle was ever in scope; only the make was unrecognised.

One of them exposed a real bug rather than a missing name. "Avida," with a trailing comma failed to match the "avida" already in the exclusion list, because make keys were normalised for case and whitespace but not punctuation. Any feed emitting a stray comma or full stop bypassed the entire taxonomy — the exclusion lists, the aliases, the canonical makes.

Fixed at the key, so it benefits every lookup rather than that one entry. Only leading and trailing punctuation is stripped, and the character class is explicit rather than [^a-z0-9]: internal characters carry meaning in "Mercedes-Benz", and a blanket strip would eat the leading letter of "škoda", which is a real alias in the list. Both verified, along with " Toyota. " resolving to Toyota.

Seventeen caravan and camper brands added, plus Mitsubishi Fuso as a bus and truck division named in full.

Elite, Empire and Western are deliberately not added. Each is a real caravan or truck brand here, and each is a generic enough word that a car marque could plausibly use it — the same reasoning that kept Golf out. They stay in the queue for an individual decision, which is what a queue is for.

Newsletters are not listings either

06:46 am AWST · 6c04da2

Cargaz spent a fetch on /e-news/winter-2026-e-news-edition/4171/ — a newsletter, correctly parsed to null. Same class as the press releases excluded earlier: a dated article URL supplies the four digits the listing matcher wants, and "e-news" was a spelling the earlier fix did not cover.

Adds /e-news/, /enews/ and /newsletter/. Checked before adding, as with the

Add a monthly donation alongside the one-off

06:46 am AWST · be69f5f

Stripe refuses a donor-chosen amount on a recurring price outright: "you may only specify one of custom_unit_amount, recurring". So monthly is $5 a month per unit with the quantity adjustable at checkout, which still lets the donor set their own level. The page says so instead of implying the two flows behave the same way.

One-off leads as the filled yellow button because it is the lower friction ask; monthly sits beside it as an outline so the pair reads as an option rather than a decision to agonise over. Both are hosted Stripe pages, so wallets work and the app still holds no Stripe credential.

Cards and wallets only, Managed Payments disabled, SPOTLOT on the statement via the shared product, and a confirmation that mentions cancelling any time.

Remove the embedded checkout; the hosted page is the better donation

06:46 am AWST · 41bfe37

Taj called it, and the deciding reason is Apple Pay. Wallets only appear where the hosting domain is registered with Stripe as a payment method domain, so an embedded form loses them while the hosted page has them by default. For a donation, one tap almost certainly beats typing a card number, and that outweighs staying on the domain.

The rest agrees. The embed cost two npm dependencies, a server action, a client component and a live secret key in the environment, and it took four version-specific failures to get working. The hosted link is one public URL in an env var. Now the Spotlot Stripe account is themed, the hosted page is dark with a yellow button carrying black, so it looks more like Spotlot than the embed did with a white form card in a dark panel.

Both Stripe keys are deleted from Vercel and the placeholders from .env.local. This application now holds no Stripe credential of any kind: the entire integration is a URL.

Embed the donation form in /support

06:46 am AWST · 77ea3e3

Four failures got us here, each only visible once the one before it was fixed, because Stripe validates in order and none of it could run without a live key:

1. Managed Payments is on by default on the new account and cannot collect donations. It refused custom_text here exactly as it refused submit_type "donate" on the payment link. Disabled per request, which also keeps both donation paths identical with Spotlot as merchant of record. 2. ui_mode "embedded" is retired on this account's API version. 3. ui_mode "form", which the newer docs describe, is not valid either. This account accepts hosted_page, embedded_page or elements. 4. The session was requested inside a useState initialiser, so it ran during render. Next refuses that ("Server Functions cannot be called during initial render") and the deploy failed at prerender. It now runs from Stripe's own fetchClientSecret callback after mount, which also keeps /support static.

The local build could not have caught the fourth, because without the Stripe keys the embed branch never rendered. web/.env.local now carries obvious placeholder keys so a local build exercises the same path.

The container carries a minimum height. The frame measures itself and posts a height back, so a container sizing to its content gives it nothing to grow into and collapses it to zero, which is what the first working session rendered as.

A typo, a forklift and five caravans walk into the review queue

06:46 am AWST · 2bd7dbd

Eight aliases arrived from the newest sources and they split three ways, which is the reason the queue exists rather than a rule.

Six are out of scope: Apache, Ever Green, Villa Caravans, Viscount and Winnebago Industries are caravans and RVs, and a Komatsu forklift came through a dealer's feed alongside its used stock.

One is the opposite failure. "HYUNDI GETZ" is a dealer's misspelling of Hyundai, and left unmapped it drops real Hyundais out of the corpus entirely — the same queue, the same shape, the exact inverse of what to do about it. Mapped to Hyundai, along with "hyundaii".

One is a make we were missing. ZX Auto is a Chinese ute maker; its Grand Tiger arrived with body type "c/chas", which is a vehicle a dealer trades. Added as canonical rather than excluded.

Verified all three groups plus controls: HYUNDI and ZX AUTO resolve to real makes, the six caravans and the forklift are out, and Toyota, Isuzu and Golf are untouched.

Banner: the contour dome, from Taj's reference

06:46 am AWST · ef05629

Replaces the trend line with the reference Taj uploaded — a stack of nested wavy lines forming a hill.

The whole character comes from one move: every line is the same ellipse shifted down by a fixed spacing and clipped to the canvas. Along the steep left flank each lower line exits the bottom edge slightly further right, which is the fan; and the lowest lines survive only as short arcs near the peak, because the rest of their ellipse is already off-canvas. Undulation phases drift slowly with the line index instead of being independent, so contours stay roughly parallel the way real ones do — independent phases made neighbours cross, which read as noise. Seeded, never Math.random(), so the asset is reproducible.

Colour departs from the reference deliberately. The reference is white on black; here the stack recedes into the black and exactly one line is spot-yellow-hot. That is BRAND.md §5 — yellow is the subject, grey is everything else — the product's own chart grammar as ornament, and the reason it reads as Spotlot's rather than as a generic contour pattern. An all-yellow stack was rendered and rejected: no subject, and BRAND.md warns acid yellow at scale is fatiguing.

The cover now carries no type. The reference has none, and the company page prints the name and tagline directly beneath it, so a tagline here is the same sentence twice within 100px — the argument that already keeps the wordmark off. Type over the dome was rendered and the lines run through the letters.

The personal banner takes the dome re-proportioned, not scaled: 4:1 against the cover's 6:1, and it still has to carry mark, wordmark and tagline, so the dome sits low as a horizon under the type.

check-brand-layout.ts rewritten for a design with no type on the cover. It earned itself twice: the dome's left flank was calculated to clear the logo tile and rendered 9px inside it, and the first version of the check measured a gap between two contour lines rather than above the dome, because at 2x the lines resolve as separate row-bands.

Em dashes out of the copy, everywhere a reader sees one

06:46 am AWST · db4239c

62 rewrites across 14 files, each given the punctuation its sentence actually wanted rather than a blanket swap: full stops where two statements were joined, colons where a list or restatement follows, commas for asides, and a middle dot in the label-and-value strings the valuation prints ("same badge (XLT) · weight ×1.50"). A few sentences were restructured instead, where neither a stop nor a comma read well.

Scope is what a reader sees: page copy, valuation workings and adjustment details, insight narratives, the freshness tooltips, the recall and ANCAP lines, page titles and the share card. Code comments are left alone.

Two kinds of em dash deliberately survive. The 44 bare "—" cells that stand for a missing value in a table are a typographic null marker rather than punctuation in a sentence, and every alternative reads worse. And sold.ts matches a literal em dash inside its title regex, where changing the character would stop it recognising sold cars.

One stored insight row still carried the old wording, since the narrative is persisted when the engine runs rather than rendered from source each time. Its punctuation is updated in place to match what the fixed code now generates; no figure was touched.

Support Us: one pill, one page, and the form on the page

06:46 am AWST · 035fc1e

Four changes, all Taj's calls:

The nav pill reads Support Us rather than Support, which could be read as a help desk, and it no longer rides at the end of the wrapped links row on a phone, where it landed wherever the wrap left it and looked like it was floating. On mobile it now shares the first row with the wordmark, pinned right; on desktop it sits inline after the links.

The panel under a delivered valuation is gone. The ask lives in the nav and the footer, and the valuation page goes back to being only about the car.

/support is tighter: the ask and the form share the opening panel instead of a tall block of text over a duplicate call to action at the bottom, and the duplicate is deleted.

Embedded Checkout is built and activates the moment a Spotlot key pair is set: the form renders inside /support with redirect_on_completion never, so a donor pays without leaving the site. The secret key is read only inside the server action, never exported and never bundled; if a session cannot be created the component falls back to the hosted donate link rather than showing a broken form. Unconfigured, as now, the page shows the link button exactly as before.

Em dashes are out of the donation copy, the page titles, the share card alt text and the Stripe confirmation message. Roughly 120 remain elsewhere in the app and want a careful pass rather than a blind find-and-replace.

Free means donation-funded, and the carve-out was wrong

06:46 am AWST · 63a77ea

Taj confirmed the scoping is right and settled the funding model: Spotlot runs purely on donations, with no intention of ever changing that.

So the reserved right to charge for "automated access at volume" is gone. Beyond the funding decision it was already wrong — /support has been live promising "A free API. Open to anyone who wants to build on top of it", so the About was contradicting a shipped page.

"There is no payment mechanism in the product" also goes. It was true when written and is now false: DONATE_URL is set in production and the Chip in button renders on /support today, verified against the live site rather than assumed from the code.

The mission section now says what funds it — donations, not users — and what it will never become: no paywall, no advertising, data not for sale, API free to build on, everything readable whether or not anyone donates. That language is deliberately aligned with /support, which is the canonical statement of the promise; the file now records that the two must not drift.

Wording is "there is no intention to change that" rather than a guarantee. Not hedging: it is the strongest claim that is actually true, since a company can state its intention but cannot bind a future it does not control. The sign-off note records the decision and its date.

Personal About picks up the same commitment, since that profile is where most of the early reach lands.

38 more from a re-mine — 672 sources, and the seam is thinning

06:46 am AWST · c452103

The sitemap sweep changed what there was to mine. 270 sources went from publishing one index page to publishing their whole yard, which put 20,825 new captures on disk — different pages, not the same ones re-read — and a mine over the newest 25,000 found 366 candidate hosts against 304 last time.

The yield is the honest headline: 38 of 244 that resolve, 15.6%, against 36% and 38% on the two sweeps before it. Recorded so the next person does not re-run this expecting the earlier rate. Discovery is no longer where the listings are; depth on known dealers is.

cars4us.com.au is worth its own line. It was surveyed hours earlier and parked — 884 vehicles, crawler-friendly robots, the whole inventory in one fetch, but no structured data any extractor could read. It probes clean now because the JSON-LD extractor learned to read Product nodes in the meantime. A fix made for eight broken sources quietly unblocked a ninth nobody was looking at, which is an argument for re-probing the parked list whenever an extractor changes.

Platform carried per host: 33 jsonld, 5 easycars.

Company About: what we do, why we exist, and why it stays free

06:46 am AWST · 00065bb

Restructures the company About to the three movements Taj asked for. Five rewrites were drafted independently and scored on three lenses — voice, adversarial honesty, and whether a dealer would follow the page. Voice and persuasion both picked the trade-voiced draft; the honesty pass picked a different one as the only draft it could not find a six-month embarrassment in. This ships the honest base with the best surviving lines grafted in, and drops what the honesty pass killed: "no account gate" (contradicted by our own bearer-key API two sentences later), "nobody has been keeping the record" (falsifiable, and rebutted by SPEC.md naming paid valuation providers), and a cost-base joke that stops being true with scale.

The free commitment is scoped rather than unconditional, and the file now records why. An absolute pledge was drafted and rejected: the product already ships a rate-limited bearer-key API, so "free forever, all of it" would be broken by the most predictable thing that could happen. What is promised instead is the record, the valuations, and never charging for the evidence behind a figure — with volume API access named out loud as the reserved exception. "There is no payment mechanism in the product" carries the weight as a present-tense checkable fact.

Still no figures, for the same reason the banners carry none, and now with a second reason: the row counts are large but observation began on 10 August 2026 and the history is days deep. "The record is young and deepens every day" is true now and stays true; "a day not recorded is gone" makes the youth the argument rather than the thing to hide.

Short variant corrected — it said listings sell, which the About itself says no public source confirms.

A Support pill in the nav

06:46 am AWST · 517a71b

An action rather than a nav item, so it reads as a pill — yellow carrying black, which is the one job yellow has. It points at /support, not straight at Stripe: someone should meet the context before the payment form. Hidden with every other donation surface when NEXT_PUBLIC_DONATE_URL is unset.

The ask stops arguing: no costs, no scarcity, just an open door

06:46 am AWST · 149b81c

Taj cut the cost transparency, and the reasoning is better than the framing it replaced. A bill on screen argues the tool is expensive — it isn't — and 'if the money stops, the corpus stops growing' is a threat wearing transparency's clothes. Neither was needed to ask.

So the whole financial case goes: no dollar figures, no runway, no consequence-if-you-don't. What remains is a statement and an offer — free to use, no ads, no accounts, no paywall, data not for sale; if you find it useful and would like to chip in, anything helps. The old scarcity line is now its own reassurance: everything stays readable whether or not anyone ever donates.

Same three surfaces, same restraint — footer line, /support, one line under a delivered valuation — all still dark until NEXT_PUBLIC_DONATE_URL is set. Unset, the footer states what the product is and links nowhere.

The donation surfaces, dark until there is a link to point them at

06:46 am AWST · 3f228c5

Three places, chosen so the ask is felt once and never nags: a footer line sitewide, a /support page carrying the actual bill, and one quiet line under a delivered valuation — the moment the tool has just been useful. No banner, no modal, no dismissal to out-wait.

The link is configuration, not code: NEXT_PUBLIC_DONATE_URL is a public Stripe Payment Link, so no key, webhook or secret enters this repository — which is also how a Dealerloop-owned Stripe account can be merchant of record without touching the boundary. Unset, as now, the footer offers the costs instead of an ask, the valuation line does not render at all, and /support says plainly that donations are not open yet. Nothing half-built ships.

The bill on screen is the verified figure only — US$10/month for the database, with the corpus size beside it — and says outright that hosting and the domain carry no number because neither is confirmed. Inventing one on a page about honesty would be the worst place to guess.

Personal profile copy, and the part of it I refused to write

06:46 am AWST · eefa6a6

Adds Taj's LinkedIn profile: headline (three options), About, the Spotlot experience entry, and skills. Kept deliberately free of sentences reused from the company page, since anyone reaching the company page from the profile reads both within a minute.

Every technical claim in the experience entry was checked against the source and the schema before being written: the SSRF guard, robots and Crawl-delay handling and per-host serialisation in crawl/fetcher.ts, the honest SpotlotBot UA and its "a 403 is recorded, not fought" rule, raw_capture retention, the taxonomy review status, and the valuation table's comparables, adjustments, confidence, days_to_turn_estimate, buy_price_recommendation and model_version columns. The API is described as versioned, authenticated and rate limited because route.ts returns 429 on the limit, not because SPEC.md says it should.

The About dates the start of recording to 10 August 2026 rather than staying vague about how long the record runs. Three days is not much, but implying more is the same failure as implying coverage, and a stated date becomes a real track record later.

No career history: I do not have it and will not invent it. The file ends with exactly what is needed to fill that gap, and flags the absence of any motor-trade credential as the biggest hole in the copy.

check-brand-copy.ts now covers all five length-limited fields.

Banner: a trend line that waves before it climbs

06:46 am AWST · 2dfc342

Taj asked for a yellow line squiggling up from the centre to just below the top-right corner, wave-like, with a dot on it. The banner's geometric centre sits inside the tagline, so the composition had to move — six treatments were designed independently and rendered before choosing.

Five of them bought room for the curve by damaging the type: tagline down to 40-48px, left-aligned, or the subline dropped. None of that was asked for. The one that ships leaves the type exactly as it was and pays for the corridor with 13px of extra lift.

The wave is load-bearing, not ornamental fussing. A line that only ever rises is the "line goes up" gesture, and on a brand whose whole claim is that it does not fabricate data that is worth avoiding even in ornament, so it moves the way a market moves and eases into the terminus. No axis, no gridlines, no ticks: it plots nothing and must not be readable as a claim about a quantity. The dot is the mark's own dot at another scale.

The personal banner gets the same device but not the same path — that canvas is 4:1 against the cover's 6:1, and the proportionally-scaled curve reached the corner as a hockey stick.

Banner rendering moves to lib/banner.ts driven by a spec, so treatments render through the same code path that produces the shipped asset; verified byte-identical on the unchanged assets before any curve landed. Both banners also render at 2x, since LinkedIn lays the cover out at 1128 CSS px and the spec-size file upscales on retina.

check-brand-layout.ts measures the rendered pixels and fails if the line comes within 15px of the type, enters the logo corner, or touches an edge. Earned: the personal banner's first curve was calculated to clear the subline and rendered with 5px between them.

Social assets: the mark that already ships, on both platforms

06:46 am AWST · eb66c59

LinkedIn and Instagram need a logo, a banner and words. All three are generated from one script rather than drawn once, so the tagline and the mark have a single source and every asset follows a change in one pass.

The mark is the one in icon.svg, the header and the share card, unchanged — BRAND.md leaves it open but code settled it, and a favicon that differs from the avatar is not an identity. The platforms take one side each of BRAND.md's inversion: black field for LinkedIn, yellow for Instagram.

No corner radius is baked in, because both platforms mask what you give them and a baked radius peeks outside theirs. The mark centres on its painted ink rather than the 48-grid — round caps and the dot push the extent off-centre, which 48px forgives and 1080px does not.

No figures on the banners. A banner sits for months, so a count printed on it is a stale count wearing confidence; the same reason they came off the share card, and it proved itself here — listings moved 96,016 to 97,421 while these were being built. brand-figures.ts prints dated counts for posts instead, and names any context table that is empty so a capability the corpus lacks never reaches a caption. vehicle_recall has zero rows and is called out in the copy as unclaimable.

proof-sheet.png renders every asset at the size each platform actually paints it: the mark holds at 32px, Instagram's circular crop clears it, and the page logo misses the banner's type.

A field named litres that holds cubic centimetres

06:46 am AWST · eaa26c9

Five TWG Cars listings failed with "vehicle insert failed: numeric field overflow" and were lost. The cause is a lying field name: Dealer Studio publishes engine_size_litres = 3283 for a 3.3-litre Mazda CX-90, and the real value sits in enginesize_litres = 3.3 immediately beside it. engine_litres is numeric(4,1), so 3283 does not fit and the whole vehicle insert failed — the listing was not partially stored, it was not stored at all.

The extractor read the mis-named field directly and had the correct one only as a `??=` fallback, which never fired because the first assignment had already succeeded.

The threshold for "this must be cubic centimetres" is 100, not 10, and that distinction is the interesting part. An Iveco S-Way at 12.9 litres and a Hino 700 Series at 12.0 are both in the corpus and both correct — heavy truck engines really are that size. A rule that flagged anything over 10 would have corrupted two real values while fixing one. Nothing on a dealer's lot has a hundred-litre engine.

Audited the stored data rather than assuming: 52,865 of 52,868 vehicles with engine data are in the plausible 0.5-10L band, the two truck values above are right, and one was genuinely wrong — a Kawasaki KLR650 recorded as 651 litres rather than 0.651. Corrected.

Four camper brands out of scope, and one that deliberately stays in

06:46 am AWST · d86295c

The review queue held five aliases whose vehicles were already excluded by body type — "caravan" and "1 axle" — but whose makes were unrecognised, so each sat pending for a human who would only have agreed with the body type. Jawa, Cub, MDC and MAN join the caravan and truck lists.

Golf does not, and that is the point of the commit. A Golf Caravans Savannah Maxxi 501 is in the corpus and is certainly not a car, but "Golf" is also exactly what a mis-parsed Volkswagen Golf would arrive as — make "Golf", model "GTI" — and excluding the make would drop those silently forever. There are no such rows today, which is precisely when the mistake would be invisible.

So that one is marked ignored as an individual alias with the reasoning attached, which is what a review queue is for. Verified after: Jawa, Cub, MDC and MAN are out; Golf, Volkswagen and Toyota are still cars.

Queue is empty — 42,671 mapped, 30 ignored, 0 pending.

Twenty failures in a row is an outage, not twenty bad sources

06:46 am AWST · 61464c9

A laptop slept for thirty minutes mid-pass and woke with a poisoned connection pool. Every source then failed on its crawl_run insert, and the pass churned through 615 of them in minutes without fetching a single page — burning the queue rather than crawling it.

That is the fix I made this morning overshooting. Before it, one transient error killed the whole pass, which was wrong. After it, nothing could stop a pass, which is wrong in the other direction and cost more: a pass that dies leaves its queue intact for the next one, and a pass that churns leaves 615 sources marked as attempted.

Consecutive failures are now counted apart from total failures, because they mean different things. One source failing is that source's problem and the pass carries on. Twenty in a row is one problem, not twenty, and continuing cannot help — so the pass stops and says how many sources it is leaving. Never-crawled-first ordering means the next pass resumes exactly there.

Migrations: sync the concurrent sessions' files to the DB order of record

06:46 am AWST · e63cb4f

The 80k wave: last per-request scans move behind the refresh

06:46 am AWST · 823661e

The corpus grew 68% in a day and everything that still computed per request crossed the line at once: the /value catalogue (a full scan shipped inside every render), model_ticker's slug pass (regex over every vehicle row per view), model_movers' live-count scan, vin_two_prices and demo_gap. All now read boards written by the 6-hourly refresh (0073-0075); movers' event join drops the view entirely — it only ever read make and model, so it joins listing and vehicle directly. vehicle gains the (make, model) index every model-scoped read had been doing without.

The refresh itself had outgrown pg_cron's 2-minute role timeout — every tick died at exactly 00:02:00 and the retry loop was itself load — so the job now sets a 15-minute ceiling in its own session (0076), leaving the role default alone.

And a page must never owe its life to one panel again: home and market fetches degrade per-panel (tolerant/soft, logged, healed on next revalidation) — both pages are ISR, so before this a single statement timeout at build time could kill the deploy.

Concurrency 16 to 24: the backlog changed shape, as predicted

06:46 am AWST · 3f9e1cc

The threshold was written down before the data arrived, precisely so this could not become a post-hoc justification: below 40% of the backlog in sources holding 1,000+ listings, raise to 24-32; above it, the answer is hours not slots.

The sitemap sweep settled it. 270 sources now discover through their own sitemap, 43,670 listings became reachable that were not before, and only 11% of that sits in sources gaining over a thousand — the median source gained 162. Against the pre-existing 29,574, the combined backlog is ~73,000 listings and about 32% head-heavy. Broad, which is the case extra slots help; per-host politeness makes the head sequential no matter how many slots there are.

24 rather than 32, deliberately. The constraint that replaces politeness is database write throughput, and that has not been measured. If a pass at 24 stays clean, 32 is the next step.

Worth being clear about what the sweep found, because "43,670 more listings" sounds like more dealers and is not. These are dealers already in the corpus whose single index page renders 21 cars while their sitemap lists the whole yard. Patterson Cheney Group: 18 held, 3,668 available. Werribee Automotive against about 530. We were not missing dealers. We were seeing 4% of the ones we had.

Trade-in anchors on the median; the counter learns to mean people

06:46 am AWST · db92013

Valuation v0.2.0: the buy price starts from the weighted median, takes the 10% margin explicitly, and prices uncertainty explicitly — a discount of a quarter of the range width, capped at 5% — with v0.1's low-anchored figure kept as a floor so nothing is ever less conservative than before. v0.1 stacked prudence three deep and collapsed on wide cohorts: the 2025 Q5 that recommended $54,000 against an $88,888 mid now recommends $68,000 against $79,990, workings on the page. Tight cohorts barely move, which is the property that matters.

The counter: ua_class is recorded at the request layer (0064) — a shared link makes WhatsApp and friends run the valuation like any visitor, and they were indistinguishable — and the exclusion is web-only (0067), because a curl UA on the key-gated API path is a customer, not a scraper. The build hour's own 40 test requests are marked internal; ?internal=1 now runs the full arithmetic with no footprint at all, verified: a dry run wrote zero rows while a WhatsApp-UA fetch wrote exactly one, classed bot, excluded from the count. 239 served · 151 distinct cars, from 280 raw.

And /value shows it: Valued recently — subject, range, when; never who — with repeats collapsed and each row linking back to a live re-run of that valuation.