Ask a flag, not a 6.9KB payload, whether the payload is empty
04:11 pm AWST · 8699f63
listing_current took 65-72s to scan, and the diagnosis on file blamed the
sha back-lookup 0057 added for deduped field payloads. It was not that.
Measured on the same warm cache, first lateral only:
count over the latest observation per listing 595 ms
the same, plus the predicate `o.fields = '{}'` 43,999 ms
The 43 seconds are `fields` itself. Payloads average 6.9KB and live out of
line — 1054MB of listing_observation's 1253MB is TOAST — so asking whether
one is empty fetches the whole value back to find out that it is not.
EXPLAIN puts 68,289 disk reads on the filter node and none on the index
scan below it. The cost lands hardest on the 155,614 rows that carry their
own content and never needed the lateral: the guard written to spare two
thirds of the corpus any work was the only reason that two thirds did any.
The resolution it guards is 3.5s of the 72s.
So the dedup stays — storing resolved payloads would add 530MB now and
~550MB per crawl pass to a 5.2GB database. What goes is the read-side
question, now answered by a boolean beside the payload. 72.3s -> 8.7s on
the original query, 1.1s warm, and the guard itself is 633ms against a
595ms floor for not asking at all.
Unchanged, not just faster: `fields` resolves to the same bytes for all
81,381 rows (fingerprint 301c746fdfc7b9bb8f8ece7579e1e83a before and
after), with the same 11,427 legitimately empty. A trigger maintains the
flag, so a payload cannot arrive misdescribed even from hand-written SQL,
and check-integrity now proves the flag and the payload agree — cheaply,
through the partial index, never touching TOAST.
price_change_count was left alone deliberately: a scalar subquery in a
view's target list costs nothing unless selected, and insights.ts and
valuation.ts do select it.
The backfill's own cost is written up in OPERATIONS.md — 155k rewritten
rows bloated the indexes past 256MB of shared_buffers and cost more than
the fix saved until they were rebuilt.
One click, not two — and it was never the transition
03:23 pm AWST · d1a7ddf
Two faults behind "nothing happens when I click it", found by watching the
network rather than reasoning about it.
The first: an async callback passed to startTransition never ran. The button
rendered, reported itself enabled, and fired no request at all. A native
click() in the console proved the code path was fine and the handler simply
was not being invoked, so it is a plain async handler with explicit pending
state now.
The second is the one that made it look dead even when it did fire. cached()
serves the stale value while it recomputes behind it, so the first render
after invalidation still carries the old computedAt — one click appeared to do
nothing and two clicks worked. The handler now makes a second pass 900ms
later, which collects the value the first started. That pass is a warm-cache
render, not another scan.
Measured on one click after the fix: 61s ago to 4s ago.
Worth recording that the browser automation's synthetic clicks never reached
React at all, which sent me looking for a bug in the handler that was not
there. The native click() in the page was what separated "the code is wrong"
from "my test is wrong".
Make the refresh button actually refresh, and cover the market panels
03:14 pm AWST · 3881203
Taj reported the button doing nothing and the age drifting past 60 seconds.
Both were real, both were mine, and the cause is the same: I verified the
control in dev, where ISR does not apply, and shipped it to production, where
it does.
router.refresh() cannot refresh an ISR page. It re-requests the path, the CDN
answers `x-vercel-cache: HIT` with byte-identical HTML, and nothing changes.
Measured on production: three consecutive requests returned age 41, 44 and 47
seconds of CDN age with the same rendered timestamp each time. The button was
asking politely for a page the CDN had already decided to keep.
The manual control now invalidates the path through a server action first, so
the following request rebuilds. Cost is bounded twice: revalidatePath only
invalidates, and the rebuild calls cached(), which holds a 60s TTL — repeated
clicks cost page renders, not repeated scans of 80,000 listings. market_pulse()
is 104ms. The action takes a path but only honours "/" and "/market", because
it is a public entry point and a caller should not choose what to invalidate.
The age drifting past a minute was the second half of the same thing. An ISR
page regenerates only when a request arrives after expiry, and that request is
served the stale copy while the rebuild happens behind it, so a 60s tick left
the age oscillating between one and two minutes. The tick is 30s now: the
first one past expiry triggers the rebuild, the next collects it. Still a
plain refresh with no invalidation, because it runs for every open tab and
forcing a rebuild per tab per tick would put load in proportion to viewers.
Also adds the readout to eight market panels — insights, state, discounting,
on-roads, two prices, demo gap, segment and make — each showing its own
cache's age rather than a shared one, because they genuinely differ.
Say how old each number is, and keep them moving
03:06 pm AWST · f779f1d
Every panel already knew its own age and none of them said so. cached() has
always returned computedAt beside the value, and its own comment says the
staleness is "bounded and surfaced" so a page can show it. Nothing ever did.
Each box now carries "43s ago" in its top-right, ticking live, with a refresh
control. The page also refreshes itself once a minute, which is what
TERMINAL-PLAN.md's framing implies — a market terminal does not wait to be
asked.
The refresh goes through router.refresh(), which re-fetches the server tree and
patches the DOM: no page reload, no scroll jump, no flash of empty panels.
Deliberately not a per-panel recompute. Each panel is a full-corpus aggregate,
so an endpoint per panel would put database load in proportion to clicks —
the exact shape of the problem that made these pages time out an hour ago.
Going through the page means ISR caps it at one regeneration a minute however
hard anyone clicks.
That ceiling is why the control states its condition instead of always
inviting a click: inside the window there is genuinely nothing newer, and a
button that appears to do nothing is worse than no button. Verified both
states in the browser — "Refresh these numbers" and enabled when stale, "Up to
date — nothing newer to fetch yet" and disabled when not. A click took the
hero from 43m to 79s old and the live count from 79,897 to 80,101 without a
reload.
Two details worth their lines. The hero's timestamp is the pulse cache, not
the freshest thing on it: the panel also shows an uncached live count, and the
honest age of a panel is its oldest source, not its newest. And one timer for
the page rather than one per panel, paused when the tab is hidden — nine
panels each refreshing the whole tree would be nine identical requests a
minute, and a backgrounded tab would ask all night for numbers nobody reads.
Market renders in a minute, not per request — and stops timing out
02:55 pm AWST · 5f5a5c4
Taj reported the site failing intermittently. Diagnosed rather than guessed,
and there were three faults stacked on each other.
**Every page was force-dynamic**, so nothing was cached at the CDN and every
visit re-rendered on a serverless function. Worse, cached() is an in-process
cache, so the cache gets *less* effective exactly as visitors arrive. Home and
Market now revalidate at 60s. That is not a freshness change — DEFAULT_TTL_MS
was already 60s, so those pages have always been allowed to be a minute old.
It makes it the same minute for everyone instead of a private minute per
instance. Pages that read searchParams (/value, /moving) stay dynamic, as they
must; /coverage is left alone because another session is editing it.
**/market loaded the entire picker catalogue to print one caption.**
catalogue_json() is 40,874 rows and 1.9MB, uncached, on every cold render, and
it had started timing out outright — which is what took the page down. It was
used for a single number: "N models in the corpus". Replaced with an RPC that
counts off the base tables.
The measurement behind that is the part worth keeping. A distinct make/model
count through listing_current takes **65.6 seconds**; the same count over
listing joined to vehicle takes **195ms**, and returns the same 1,115. The view
resolves every row's latest observation with a correlated subquery, so a full
scan runs 79,864 nested loops. Anything that scans all of listing_current is
now O(n x per-row work). Migration 0066 records that for the next caller.
**One slow panel took the whole page with it.** vin_two_prices timing out
turned Market into a 500, killing eleven working panels. The three heaviest and
most optional panels now degrade to empty — which they already render as
nothing — and log the failure instead of throwing. Same lesson as the crawler
twice tonight: handle failure at the level that can continue.
models, and all four panels render.
State in advance what would make concurrency the right lever
02:29 pm AWST · 3f3e90d
I declined to raise crawler concurrency twice tonight, first because the
database was showing statement timeouts under load. That objection is now
fixed hours ago. Leaving a stale reason standing would mean the decision never
gets revisited on its merits.
The real question is the shape of the backlog, because per-host politeness
makes the head sequential and more slots cannot help it. Measured today: 110
sources carry 29,574 unfetched listings and 64% of that sits in sources with
more than 1,000 each. So concurrency is still the wrong lever.
What changes it is the sitemap sweep now finishing, which gives ~440 sources a
few hundred discoverable listings each and spreads the backlog across many
hosts. The threshold is written down before the data arrives rather than after:
if the share held in 1,000+ sources drops below roughly 40% after the next full
pass, raise concurrency to 24–32. Above that, the answer is hours, not slots.
The hero counts our watchlist, and says so
02:22 pm AWST · 16dbc64
'79,455 cars on market now' read as a claim about the whole
Australian market — false in the too-small direction, and silent
about the thing the product actually does. The label now says whose
activity the number counts: 'cars monitored on market right now',
with the mark's own dot breathing beside it (a slow breath, not a
blink; held still under prefers-reduced-motion, and never carrying
the meaning alone — the word 'monitored' does). Ticker heroes get the
same wording; their per-model floats read the same way.
Fix the cross-view price check, which was crying wolf
02:04 pm AWST · face5fd
The health monitor alerted: "4 price change(s) span index/detail — fabricated
movement". That is entry 1 in docs/DATA-QUALITY.md and the most serious defect
class this product has. All four were false, and the check was the fault.
What it asked was whether a listing's two most recent observations shared a
view. That was a fair proxy while each source produced one view. It is wrong
once a source has both — an inline index page and a sitemap of detail pages —
because the views interleave:
index@05:28 $34,390 <- today, compared against index@23:19
detail@18:23 $34,490 <- yesterday, a different view entirely
index@23:19 $34,490
The store compared index with index and recorded -$100 correctly. The check
read the last two rows, saw index then detail, and called it fabricated. Four
alerts fired the day 193 sources gained sitemaps beside their HTML index, and
tonight's own work is what made the proxy wrong.
observeListing has always compared like with like — it selects the previous
observation with .eq("source_view", sourceView) — so the honest test is not
"did the views interleave" but "are the prices this event recorded a
consecutive same-view pair". Measured before replacing: 0 of 1,381 price
changes fail the new test, so nothing that was being caught is now let through.
Two files for one change, which is untidy and left as it happened: 0064 was
applied by pasting the function alone, so the ledger recorded it without its
comment header and sync-migrations rewrote the file to match. 0065 re-applies
it documented. Both are create-or-replace so the end state is identical, and
the tree still rebuilds the database exactly. The lesson is to apply a
migration from its file, not from a paste, or the ledger records something the
file does not say.
The scheduled crawl no longer keeps up, and the docs said it did
12:04 pm AWST · 708d54c
README claimed 47,000 listings across 181 dealer sources and one marketplace,
and that the crawl "runs on a schedule and does not need a laptop awake". The
first half is now 77,000 across 635 sources and two marketplaces. The second
half is the one that mattered: technically true, materially misleading.
The scheduled workflow affords about 34 minutes of deep crawling a day against
a 2,000-minute private-repo allowance. That was a slower way of keeping up at
190 sources. At 635 it is not keeping up at all — a 300-minute pass reaches
roughly 460 sources and does not clear the backlog they discover, so sources
are now found faster than they can be fetched.
Both files say so plainly and point at COVERAGE-CEILING.md, which has the
measured throughput and the three options. A README that overstates the
operating position is worse than one that admits a ceiling, because the ceiling
is the decision that needs making.
Refresh the marketplace composition claim, which got stronger
12:00 pm AWST · 930cbe5
The demo argued CarsGuide earns its place on composition: older, higher-
kilometre, cheaper cars than a corpus of franchised showrooms. That still
holds, but every number in it had moved, and Autotrader now says the same
thing independently.
CarsGuide is 8.2 years and 114,897 km at a median ask of $24,999; Autotrader
7.9 years and 106,154 km at $22,990; dealer sites 3.1 years, 45,263 km and
$40,900. Two marketplaces agreeing separately is a stronger claim than one
making it alone, so the slide is better than it was.
The dealer-side figure moved most — 16,368 km to 45,263 — and the reason is
worth knowing rather than papering over: the corpus was dominated by franchise
showrooms carrying near-new stock, and a night of adding wholesalers and used
specialists made it look like the actual used market.
Also corrects the out-of-scope list. Autotrader was on it, recorded as
WAF-blocked on 2026-08-10, and was re-probed as reachable on 2026-08-11; it is
now crawled and the demo said otherwise. Toyota's and Lexus's national
locators take its place there.
Retract the demo's headline slide: the discount gap was small-sample noise
11:35 am AWST · a34a073
The one-slide claim was that a 2021 Corolla sells in 25 days with 80% of them
discounted first, against a BT-50 at 72 days and 17% — the car that moves fast
being the one dealers cut. It was a good story and it is not true.
On the sales accumulated since, the discount shares are 32% and 33%. That is
no difference at all. The turn times measured on exactly the same data did not
So the finding that survives is about time, not discounting, and the slide now
says only that. The retraction is left on the page rather than quietly edited
out, with a note to volunteer it if a dealer asks — a product whose pitch is
that it refuses to overstate should be visibly willing to withdraw its own
headline.
Corpus figures in the opening paragraph updated too: 77,300 live listings
across 635 dealer sources plus two marketplaces, 214,000 observations, 1,248
price changes, 919 confirmed sales, 112 makes, 1,150 models — four to seven
times what the paragraph carried yesterday. Field coverage slipped one to four
points, which is recorded as expected rather than hidden: the sources added
overnight are small franchise sites that publish less per listing than the
large yards.
Widen the index-page catch to the whole page, not just the fetch
08:54 am AWST · c6e5152
The previous commit caught politeFetch and claimed the index loop was
resilient. The very next run disproved it: CarsGuide threw from
storeRawCapture instead — "raw_capture insert failed: TypeError: fetch failed"
— left the try block the same way, and recorded found=0 after 13 pages again.
That is the same bug twice in an hour, and the lesson is about the fix rather
than the code. I patched the line that happened to fail instead of the class
of failure, so the second failure looked new and was not. Everything on that
page can throw: the fetch, the capture insert, the sitemap parse, the
extractor. The whole body is inside the catch now, which is what "handled per
page" has to mean.
Verified structurally rather than by assertion — storeRawCapture and
extractor.discover are both between the try and the catch.
Behaviour is unchanged otherwise: the page is skipped, everything already
discovered is kept, and `truncated` stays set so a partial traversal still
refuses to infer a delisting.
One unlucky packet no longer ends a five-hour pass
08:51 am AWST · 53bbeed
The chained pass died 80 minutes before its deadline. The cause was a single
transient "TypeError: fetch failed" on one source's crawl_run insert, which
threw out of the worker, rejected Promise.all and exited the process while
fifteen other workers were mid-source. Fourteen run rows were left open and
the rest of the queue was never reached.
crawlSource already handles its own per-page failures carefully — it is the
failures *outside* that handling, on the run bookkeeping, that had nowhere to
go. With 635 sources and a 300-minute pass, the chance of one unlucky packet
somewhere is close to one, so this was not an edge case; it was a matter of
waiting.
Caught per source now, logged with the source id, and the pass continues.
Counted rather than swallowed: a pass where many sources fail this way is a
broken pass, and the tally prints at the end so it cannot pass for a clean run.
Same network blip also took recanonicalise, enrich and compute-insights, which
run after the crawl in the same script — those already exit non-zero and are
re-run by the next pass, so they are left as they are.
One slow index page no longer discards a whole run's discovery
08:28 am AWST · 4387bd3
Both marketplaces just recorded found=0 after fetching 120 and 124 pages.
CarsGuide timed out on /buy-a-car and Autotrader on /for-sale/used, and
because the index loop had no per-page catch, the exception left the try block
before listings_found was computed. Every URL discovered across 120 facet
pages went with it — roughly 10,000 listings of the unique marketplace
inventory this pass existed to collect.
The detail loop has always caught per-URL and carried on. The index loop had
no equivalent, which made it the fragile half of the same pass, and fragile in
exactly the wrong place: the more facets a source has, the likelier one of
them times out. CarsGuide now has 92 and Autotrader 55, so widening discovery
this morning made this failure near-certain by this afternoon.
Caught per page now, recorded, and the traversal continues. `truncated` is set
so a partial pass still refuses to infer any delisting — same treatment as a
non-200, which was fixed for the same reason a few hours ago and only covered
half the problem.
Spend the budget where it buys cars we do not have
08:10 am AWST · eda4b90
Seven more capped sources get a raise, chosen on how much of their stock is
already held elsewhere rather than on how big they are: 2% for Maughan Thiem,
12% for Keema, 18% for Q Automotive, 0% for Village Motors and Adelaide
Vehicle Centre. Budget spent there buys new cars; budget spent on a marque
site that is a subset of its group site buys none.
Deliberately modest, because a per-source cap does not create crawl time — it
moves it. Sixteen concurrent slots over a 300-minute pass is roughly 48,000
fetches however they are divided, and the pass is already near that.
Which is the finding worth recording: for the two biggest unfetched pools the
per-source cap is not the constraint at all. Autotrader now discovers 8,974
listings against 228 before the facets were expanded, CarsGuide 2,014 against
224, and both have a 2,500 budget they did not reach — they ran out of pass,
not allowance. Neither overlaps the dealer corpus at all, so that is 10,000
listings of genuinely unique inventory waiting on crawl hours rather than on
configuration.
28% of sources may be duplicates — measured, not yet acted on
07:59 am AWST · 692455b
A dealer group runs a site per marque and usually a group site that lists the
whole network's stock, so a marque site can be a strict subset of one we
already crawl. 180 of 637 sources holding stock are 95% or more contained
within a larger source, 12,066 listings between them.
If that holds at full depth it is 28% of the crawl budget spent re-reading
known cars, which bears directly on where the crawler should run.
Deliberately not acted on, and the reason is the point. Containment is
measured on what has been fetched, and most of those sources are budget-capped
— Motorama Chery is 29% covered, Motorama Jeep 41%, Salters Cars 37%. Their
fetched cars all appear in the group site; the five hundred unfetched ones
might not. Disabling a source on a 29% sample is the same class of error as
every entry in DATA-QUALITY.md: a partial result read as a whole one.
Three are fully crawled and do hold: Scarboro Toyota is 99.7% inside Regent,
CQ Used Car Superstore 99.7% inside Reef City Motors, Westpoint Autos LDV
100% inside Westpoint Autos. Even those are left alone until the rest can be
judged on the same evidence.
Worth noting this is no longer a correctness problem — valuations dedupe
comparables by VIN as of today — only a question of what the fetches buy.
Share card: the numbers come off
06:47 am AWST · 4148a6a
Taj's call, and the right one: every platform caches a scraped card,
so a 'live' figure on it is really a stale one wearing confidence.
The pitch does not date; the numbers live on the site. With the
database read gone the card is fully static and prerenders at build.
The share card, stripped to the hero — and set in the brand's own type
06:42 am AWST · 5205973
The first cut carried an illustrative scatter and curve; Taj called it
mess, and he was right. Now it is the site's hero exactly: the mark,
the pitch in Inter Tight ExtraBold, the quiet subline, and the two
live figures in the same JetBrains Mono the terminal uses. Fonts are
static TTFs from Google Fonts (OFL) under web/assets/og-fonts, read
at render — the OG renderer cannot consume next/font's woff2 subsets.
The share card: the product, in one image
06:36 am AWST · 34381b1
opengraph-image.tsx renders the link preview with next/og: the mark,
the pitch, and the plotted point — grey comparables, one yellow line,
one yellow dot marking the spot. The figures are read live from the
corpus per scrape (force-dynamic, 2.5s timeout), so a shared link
carries the real counts at the moment it was shared — and if the read
fails the card ships without numbers rather than inventing any. Full
OG and Twitter metadata with metadataBase and a title template; the
motif carries no axes or scales, so it is brand art, not a data claim.
Every zero on /coverage now means something specific
06:32 am AWST · 8438c30
After the expansion to 639 sources, five held no listings. A zero should
always have a reason, so each was re-probed rather than left looking like a
broken crawler.
Townsville KGM was mis-seeded and is repaired: it is a Dealer Studio site
serving /vehicles, and it went in as jsonld pointed at /stock, which 404s. The
sweeps take the platform from the probe, so this is the error shape to expect
when a later sweep assumes the path a probe found — cheap to check, and the
alternative is a permanent zero.
Four are withdrawn. Three answer 404 on every inventory path we know and
publish no used-vehicle link that even a deliberately naive search can find.
The fourth, Autosports Group, answers HTTP 504 to everything on two probes
hours apart — that is a server fault rather than a refusal, and the note says
so, because it is worth re-probing another day rather than writing off.
Also confirms the tightened URL patterns are not over-filtering: searching
those pages for anything matching "used or demo, plus a four-digit number"
finds nothing. The sources are empty, not hidden.
635 dealer sources enabled.
Mobile: counters above the nav, and charts scroll instead of shrink
06:27 am AWST · f5447c2
On a phone the header now stacks logo → counters → links; the pulse
reads before the menu, per Taj. The three wide SVGs (ticker scatter,
Today's Range, the dealer sales strip) rendered their labels at ~5px
when scaled to 375px — now they keep a 560px minimum inside their own
horizontal scroll, per the house rule that wide content scrolls
rather than implying legibility it does not have. Audited all nine
routes at a true 375px layout: zero horizontal overflow.
One car, one comparable — the same answer with an honest n
04:53 am AWST · 4767964
A dealer group runs a site per marque and often a group site too, so the same
physical car is advertised more than once. 18.6% of live vehicles in the
corpus are on two or more sources, and a Toyota HiLux cohort of 1,283 listings
covers 1,065 distinct cars. Every one of those duplicates was voting twice on
the median and pulling the range towards whichever dealer runs the most
shopfronts.
This became worth fixing tonight because tonight made it worse: seeding
{group}{marque} and {suburb}{marque} networks deliberately adds the sibling
sites that carry the same stock.
Safe to merge because vehicle identity is VIN-based here. A car without a VIN
gets a fingerprint scoped to its own source, so two sources can never be
collapsed on a guess — only positively identified duplicates merge. The copy
kept is the most recently observed, and the cheaper one when both were seen at
the same moment, since that is the price a buyer could actually pay.
Measured before and after on four cohorts. Comparable counts fall 14–33%
(HiLux 369→319, CX-5 208→140, Ranger 366→249, i30 168→128) and every midpoint
moves less than 1% — 52,109→52,232, 25,716→25,751, 49,996→50,245,
17,552→17,423. So the duplicates were not systematically skewed; they were
inflating the sample. The Ranger's upper bound widens 61,111→65,374, which is
the badge spread showing honestly instead of being compressed by repeats.
Known and not changed here: the comparable query takes .limit(400) with no
order, so a 1,283-listing cohort is an arbitrary 400. That is a separate
question needing its own measurement, and it is not made worse by this.
109 more, and the point where this method runs out — 639 sources
04:36 am AWST · b8edbd8
Two extensions of sweep 7: the suburb threshold drops from 140 listings to 25,
reaching 210 towns rather than 70, and the marque-first order is added because
the corpus already contains skodaperth, cheryhobart and cherybunbury. Chery
alone runs at least 26 sites as chery{suburb} — Adelaide, Blacktown, Castle
Hill, Coffs Harbour, Cranbourne, Dandenong and on — which suburb-first would
never have found.
The yield curve is the part worth recording. Sweep 7 resolved 18% of its
guesses in DNS and 38.5% of those held a car. This one resolved 3.9% and
29.5%. The strong suburbs and the obvious pattern went first, as they should.
A third round of the same idea is not worth its fetches, and I am recording
that here rather than running it to find out twice.
Platform is carried per host rather than assumed: 107 jsonld, one imotor, one
dealer-studio. Hardcoding it — which the last two sweeps did get away with —
would have pointed the wrong extractor at two working sites.
State is set for 82 of 109 from the corpus and unambiguous town names; the
rest are null rather than guessed.
156 more from {suburb}{marque} — 530 dealer sources
04:03 am AWST · 3a621d3
The largest single sweep so far, and the best hit rate: 38.5% of the domains
that resolve.
Franchise dealers here name their sites after the town and the marque —
berwicktoyota, cardiffhyundai, cranbournekia. The only open question is which
towns to ask about, and the corpus answers it: these 70 suburbs are where
listings we already hold are physically located, so every one has proven
franchise trade rather than being a name off a map.
2,355 candidates, 437 resolve, 405 new, 156 hold a priced car.
The order of the filters is what makes it affordable, and worth stating
because this is domain guessing, which failed twice before. DNS rejects four
in five before any HTTP request is made. The probe then requires a real car
with a price on the page, so the 249 that resolved without stock — parked
domains, redirects to a group site, franchises listing new models only — were
refused rather than seeded and left at 0% on /coverage forever. Last night
that check did not exist and four such sites were seeded.
States are taken from the corpus rather than inferred from the name: the modal
state of listings already held in that suburb. All 156 resolved.
Guess {suburb}{marque}, with the suburbs taken from the corpus
03:57 am AWST · 8a10c13
The other dominant naming pattern for Australian franchise dealers, after
{group}{marque}. Brighton, Werribee and Traralgon each turned out to run most
of their marques this way, so it is worth asking the question directly.
The suburbs are not guessed. They are the towns where listings we already hold
are physically located, so every one has proven franchise trade rather than
being a name off a map. 70 suburbs with 140+ listings gives 2,355 candidates.
DNS before HTTP, as with the sibling guesses: 437 of 2,380 resolve, and 405
are new. Four in five guesses cost a failed lookup rather than a robots fetch
and fourteen path probes.
The first version of this printed zero guesses and did not error. It asked for
.limit(50_000) and got 1,000, because that is PostgREST's cap however large
the limit — so it counted 2% of the corpus and no suburb cleared the
threshold. Fifth time that cap has silently truncated something in this
codebase, and the first where the symptom was an empty result that looked like
a real answer. Paged by key now.
32 more, by completing franchise networks — 375 sources
03:36 am AWST · ea7995a
Werribee gains six marques, Brighton eight, Traralgon five, Bartons four.
Every one confirmed to hold a priced car before seeding.
The method is worth judging on its own record, because guessing domains has
failed here twice. Inventing them from group names ran at 35%, then zero, then
zero. This ran at 36% of the domains that resolve — but the real difference is
cost, not rate: DNS rejected 589 of 677 guesses before a single HTTP request,
and the 56 that resolved without holding stock were refused by the probe's
priced-vehicle check rather than seeded and left at 0% on /coverage.
So the honest framing is not "guessing works after all". It is that a guess
made from a pattern already crawling, filtered by DNS and then required to
produce a car, costs little enough to be worth making.
Complete the franchise networks instead of guessing at names
03:31 am AWST · 65aea93
Australian dealer groups run one domain per marque — duttonsford,
duttonsnissan, duttonssubaru — so once a group's pattern is visible in domains
we already crawl, the marques missing from it are a targeted guess.
This is not the method that failed before. Inventing domains from group *names*
ran at 35% and then twice at zero, because the name was a guess and so was the
shape. Here the shape is read off seven existing Duttons domains, seven Reef
City, seven Motorama, five Traralgon; only the marque varies.
17 groups have three or more marque sites, which yields 677 sibling domains.
Guessing is only done on the TLD a group actually uses — a group on .com.au is
no evidence for the same name on .au.
DNS first, HTTP second: 88 of the 677 resolve, so 589 wrong guesses cost a
failed lookup instead of a robots.txt fetch and up to fourteen path probes.
The survivors go through the same probe as everything else, which now requires
a priced car on the page, so a wrong guess that does resolve still cannot be
seeded.
Brochure and accessory paths, which the four-digit rule cannot catch
02:19 am AWST · 34e927d
Cumberland LDV spent 13 fetches on /new-vehicles/ldv-g10-van/, GWM Ryde 21 on
/accessories/ and /our-stories/, Village RAM 8 on /new-vehicles/. All parsed
to null, correctly, and all counted against those sources' parse rate.
The four-digit rule added earlier does not catch these, and RAM shows why:
its model names are 1500, 2500 and 3500, so /new-vehicles/1500-big-horn-crew-
cab/ carries four consecutive digits without carrying a car. A path rule is
needed as well as a digit rule.
Measured before adding: /new-vehicles/, /accessories/ and /our-stories/ hold
zero of the 1,426 listings currently held from JSON-LD sources.
Deliberately not excluded, though it would have raised the same numbers:
/demo-cars/for-sale/ and /new-cars/for-sale/ hold 64 real listings. Llewellyn
GWM fails on those pages for a different reason — the site publishes no vehicle
JSON-LD on them at all, only Organization and WebSite — so null there is the
extractor reporting the truth about the page, and excluding the path would
have hidden a real limitation while losing 64 cars elsewhere.
"news" matched the listing pattern because "new" plus an s does
01:49 am AWST · 274257d
Rockdale City Chery spent 47 of its 181 fetches on press releases about award
wins; Lilydale Hyundai spent 25 on /click-to-buy/ configurator pages. Both
parsed to null, correctly — there is no car on a press release — and both were
counted as extraction failures, so two working sources read as 73% and 85%
parsed.
The cause is in the matcher's own wording. Its category alternation contains
`new` with an optional `s`, so "latest-news/" matches it exactly, and a dated
article URL supplies the four digits the rest of the pattern wants. Nothing
about the page had to look like a car.
Excluded now, along with blog, article, review, award, click-to-buy, our-team
and careers. Checked before adding rather than after: none of the 1,426
listings currently held from JSON-LD sources has a URL matching any of them,
and seven URL shapes were tested both directions — the press release and the
configurator are out, a Ranger, an X-Trail, a Duster and a HiLux stay in.
Raise detail budgets, now that the budget is what binds
01:42 am AWST · 1446272
Twelve yards go from 1,000 to 2,000 or 2,500 detail pages a pass.
Timing is the point. Raising these yesterday would have bought nothing: the
rotation read a source's first 1,000 listings and treated everything past that
as never-seen, so at a yard over that size the queue filled with cars already
held and a 1,000-page budget spent 200 pages on new ones. The number was never
the constraint.
With the rotation fixed, the same sources use the whole budget — Bartons 221
extracted to 1,021, Westside 200 to 1,000, both landing exactly on their cap.
That is what makes the cap worth moving.
Sized against measured throughput, not guessed: 5.6–6.2 seconds a page at
these hosts, so 2,000 pages is ~3.3 hours and 2,500 ~4.2, both inside a
300-minute pass. One source holding one of sixteen concurrent slots for that
long costs about 5% of the pass.
Committed rather than left in the database, for the same reason the facets
Correct the fetch rate: overhead was dominating the first measurement
01:41 am AWST · fe7e3b2
This document reported 9.9, 10.1 and 10.3 seconds per page as the measured
crawl rate. Those came from runs of ~220 pages, where a pass's fixed costs —
index traversal, the delisting check, its own bookkeeping — are spread over
few fetches and dominate the average.
The long runs that followed put the marginal rate at 5.6 to 6.2 seconds for
most large yards, and 9.6 for a slower group. The clearest evidence is the
same host twice: Westside measured 9.9 s/page over 222 pages and 5.95 over
1,022. The rate never changed; the overhead stopped dominating.
That halves the 80-hour figure to roughly 48, which matters because this
document exists to inform where the crawler runs.
The ceiling is nearer than the arithmetic said, measured
01:18 am AWST · 1f5fb68
The 28,817 figure was taken while the pass carrying the rotation fix was still
running, so it measured a corpus the fix had not reached yet. One pass later:
Bartons +780, Country Cars +761, Brisbane Cars +701, Westside +596, John
Hughes +369. Every earlier pass had added roughly nothing to these.
Bartons is now within 48 listings of its sitemap and Westside is past its
count — both effectively complete, on the two yards this started from.
The correction that matters: the binding constraint was never only wall-clock.
Most of the clock was being spent re-fetching cars already held, so the hours
looked scarcer than they were. At ~700 per big yard per pass the current gap
closes in two or three passes, not days. Crawl-delay still sets the floor and
the head is still sequential; the head now moves.
Stop the sitemap estimator counting model brochures as stock
01:11 am AWST · 372509b
A sitemap entry shaped like a listing can be a brochure. /new-vehicles/
t-cross/ and /specials/1500-rebel/ have a nested path and a model name and no
car on them, and four franchise sites were counting 36, 14, 12 and 10 of those
as inventory they hold none of — inflating the estimated upside by their whole
sitemap.
Found by checking rather than assuming. Those four sites were withdrawn
earlier tonight as having no used stock, then the sitemap pass turned round and
applied a sitemap to one of them, which is a contradiction worth chasing.
Opening every URL settled it: all 72 across the four are /new-vehicles/ or
/specials/ pages with no price. The withdrawal was right and the estimator was
wrong.
Since the 28,817 figure in COVERAGE-CEILING.md rests on this estimator, the
four largest contributors were audited directly: zero brochure-path URLs
between them, and sampled pages open as priced cars. That number stands, and
the document now says it was checked.
The crawler itself was never affected — it rejects brochure URLs at discovery
on the four-digit rule, so none of this reached the corpus. Only the estimate
was wrong, and an estimate is the whole point of that document.
Measure the coverage ceiling: 28,817 listings known and not fetched
12:38 am AWST · 10dbda6
Until now "how much are we missing" had no honest answer — a source stuck at
200 listings looked the same as a small dealer. Asking every large source for
its own sitemap answers it.
Across the 113 largest sources: 107 serve a usable vehicle sitemap, we hold
46,983 listings across them, and those sitemaps advertise 71,183. The gap is
28,817, concentrated in 31 sources and mostly in ten — Brisbane Cars is short
5,321 on its own, Country Cars 3,709, Booran 2,588, John Hughes 2,385.
The gap is not discovery. Every one of those URLs is known. It is fetch rate,
and the rate is deliberate: measured at 9.9, 10.1 and 10.3 seconds per page
across three sources on one pass, which is the Crawl-delay those hosts declare
and SPEC binds us to honour.
The unpleasant part is that concurrency cannot fix it. Politeness is per host
and the gap is concentrated in a few hosts, so Brisbane Cars alone is ~15
hours of necessarily sequential fetching. Concurrency helps the tail; the
missing inventory is in the head.
Records what was ruled out and why, so it is not re-explored: the index pages
render 21 priced cars per fetch and would be 21x cheaper, but ?page=2, ?p=2
and ?offset=21 all return 200 with the same 21 listings — pagination is
client-side, verified rather than assumed from the existing comment.
This feeds the open decision about where the crawler runs, because hours per
day is exactly the constraint.
84 more dealer sites, every one opened before it was trusted
12:23 am AWST · 7306758
343 dealer sources, up from 259. Found by mining 20,000 stored captures for
the sites our sources link to, then probing each.
Two changes made this sweep bigger than the ones before it. The miner had only
ever matched .com.au, so .au, .net.au and bare .com dealers were structurally
invisible to the best discovery method we have — widening it turned 178
candidate hosts into 304, and lismorecars.au here is a site no earlier sweep
could have proposed. And the probe now opens discovered URLs and requires a
real car with a price rather than counting links, which is why 84 of 250
new models, as it did four times last night.
166 candidates did not pass and are not here. Most publish no inventory at any
reachable path; some answer a WAF challenge, recorded and left alone.
Heavy on franchise networks one group runs under many domains — Duttons across
eleven marques, Reef City across eight, Tynan across eight, plus Bartons,
Buckby, Von Bibra, Patterson Cheney and Rex Gorell. Each is a separate site
with separate stock, so each is a separate source.
Names are split on marque tokens rather than left as the domain stem, because
they appear on /coverage and on each dealer page: "Bartons LDV", not
"Bartonsldv". State is set only where the domain names a town — 29 of the 84 —
and left null otherwise rather than guessed, since the listings carry
postcodes.
Empty the review queue: two bikes, a campervan and an Omoda
12:20 am AWST · f478d7a
The taxonomy review queue is meant to hold things a human still has to decide.
Four entries were sitting in it and none of them needed a human.
"BMW Motorcycles" K03 and KA2 are the motorcycle division under the name one
dealer's feed uses. "BMW Motorrad" was already excluded and this spelling was
not, so two bikes waited on review while the rule that covers them existed.
Named explicitly, never the parent — BMW's cars stay in scope.
"Interstate ONE CAMPERVAN" joins the caravan and motorhome list.
"OMODA JAECOO O9 PHEV" is the interesting one. The combined-marque resolver
knows OMODA*, JAECOO*, J-numbers as Jaecoo and C/E-numbers as Omoda, and
returned null for O9 rather than guessing — which is exactly what it should
have done. O5 and O9 are Omoda's own numbering, so the rule is added because
the answer is now known, not because the refusal was wrong.
Verified all four plus two controls: BMW and Toyota are still cars, and an
unrecognised OMODA JAECOO model still returns null for review.
marked out of scope. Queue is empty.
Withdraw four sources that had no used inventory, and stop guessing
12:12 am AWST · d662584
Carlin & Gazzard LDV, RAM and Volkswagen, and Village GWM Haval: real
franchise dealers whose sites list new models only. Their stock pages offer a
dozen links that look like inventory and every one is a brochure —
/new-vehicles/ldv-t60-max-ute/, /new-vehicles/id-4/. The crawler extracted
nothing from any of them and they would have sat at 0% on /coverage forever.
Disabled with the reason rather than deleted, so the survey is on record.
I seeded them on link count, which is the same mistake mazda.com.au only
avoided because I opened it by hand. The rule should not be that someone
happened to look, so the probe now opens discovered URLs and requires a priced
car before calling a host usable.
Two corrections to that check, both found by it disagreeing with the crawler,
which is the useful direction for a disagreement:
Checking only the first discovered URL called Llewellyn GWM a brochure site
while the crawler was extracting seven real cars from it — dealer stock pages
routinely lead with a promotion or a current-model tile before the used cars
start. It tries three now.
/our-stock is in the guessed-path list, seen on four sites in one night. The
homepage fallback finds that shape sometimes and not reliably, and without it
the probe reported a working source as having no inventory page.