Sitemap: take the generic matches too when the extractor finds few
11:45 pm AWST · 93c9cbb
Correcting a fix that did nothing. The previous commit taught
sitemap-scan.ts carma's /product-detail/N shape and claimed that fixed the
crawl. It did not: crawl.ts imports only sameSite from that module and
carries its own sitemap filter, so the change reached probe-candidates and
probe-sitemaps and never touched the crawler. The commit message was wrong
about what it did.
The real mechanism is in listingUrlsFromSitemap, which asked the extractor
first and returned its answer whenever it was non-empty. The design handles
"extractor finds none" — that is what the fallback exists for — and not
"extractor finds a few, and the wrong few". carma publishes 817
/product-detail/N cars and 626 /used-cars/{model} browse pages; the JSON-LD
extractor recognised five browse pages; five being more than zero meant the
fallback never ran, and the crawl reported 5 found, 0 extracted while the
yard stayed invisible.
rule's, so a precise extractor is untouched. When the generic rule finds
substantially more, the two are merged rather than swapped, because the
extractor may hold URL shapes the generic rule rejects. The generic rule is
the conservative one already — same host, not a known non-listing path,
nested, four-digit id or year in the last segment — so merging cannot admit
anything the crawl would not already accept from a source whose extractor
found nothing.
Found because Taj asked whether anything was still waiting to verify the
earlier fix. Nothing was: I had queued a crawl, not checked it, and said the
work was done.
Four findings where there was one, and the panel stops repeating itself
11:29 pm AWST · 9ffa3db
days." The history says no, and worse than rarely: across 29 recomputes since
12 August, market_insight_history holds exactly one kind. Orange was not
merely winning, it was the only hypothesis that had ever cleared the bar
once.
The cause was the library, not the bar. Three of the four hypotheses could
not clear: the regional spread is real but explained by age and kilometres
once controlled, and both day-count tests fail their three-day effect floor
because days_on_market starts at our first observation and the window is a
week. So the panel was structurally frozen, and a hero slot that says
"surfaced automatically" while showing the same sentence for six days is
worse than an empty one.
What the method could see was the constraint nobody had written down: a
residual is mean-zero inside its own cohort, so only attributes that VARY
within a make and model are visible to it. Colour does. So do transmission,
drivetrain and condition - a Ranger yard holds manual and automatic, 4x2 and
4x4, demo and used - and fuel does on the models sold both ways. Body type
does not, which is why adding it would have returned nothing.
So four hypotheses join the library on the existing residual machinery, and
the run went from one finding to four: orange at 7.9%, four-wheel drives at
6.3% over the two-wheel drive of the same model, demonstrators at 6.0% over
used, hybrids at 8.8% over petrol. All four at p < 0.001 with six figures of
n.
A wider library is more chances at a false positive, so the bar went up with
it rather than down: every surviving p is now multiplied by the eight
hypotheses the run attempted, on top of the within-test correction each
already carried, and both numbers appear in the workings as separate rows so
the eyebrow and the panel cannot disagree. Nothing that survives that was
marginal.
The page features one finding and lists the rest, rotating by UTC day.
Deterministic, because every finding shown has cleared the same bar so which
one leads is presentation rather than ranking, and a reader who returns
tomorrow should see the panel move without anything being reordered behind
them. The eyebrow says "1 of 4 today" from the first day, since the retested
line hides until a finding has survived a second run.
Recompute cadence is unchanged and already correct: cron-crawl.sh and the
Actions crawl both run compute-insights, so "retested after every crawl"
stays true of the new four.
Seed automatically in the discovery chain
11:19 pm AWST · bbdc27e
Seeding was the one step in the whole pipeline that waited for a person, and
the reason it waited has gone away.
"Seeding stays a decision" was written when the probe verdict read
`platform, url, N links` — a statement about whether an extractor recognised
a page, and nothing about whether the yard was worth having. That needed a
human because the evidence did not answer the question being asked.
The verdict now carries what the decision turns on: three priced cars read
off the stock page, the dealer's name and suburb, and two checks that used to
be the human's. It refuses a host whose dealer name a `dealer:` source already
publishes, because a second domain for a yard we crawl adds copies rather than
cars. And it refuses a host whose inventory URL lands on a source we already
have — the way waratahautocentre.com.au came within one crawl of duplicating
newlambtonautocentre's entire inventory today.
This is not a lower bar. It is the same bar, written down and checkable.
What still wants a person is the judgement the probe cannot make: whether a
dealer belongs in the corpus at all. The review queue and taxonomy_status
carry that, and nothing here touches either.
The last chart with labels in the box, and a tape that stopped hiding its yards
11:14 pm AWST · 51ebc88
Asked to give /moving and the home page the treatment the valuation and model
pages got, and they do not need it: neither page has an SVG chart. Their
overflow-x-auto wrappers hold tables, and their bars are divs. An audit over
every page settles it rather than trusting a grep - the only chart still
carrying text inside a width-scaled viewBox anywhere on the site was the RBA
rates strip on /reports, which is now converted like the rest and leaves zero
scaled SVG text nodes across /, /moving, /reports, /market, /coverage, /api,
the model pages and the valuation.
/moving genuinely needed nothing: no page overflow, no clipped text, one table
scrolling inside its own container by design, and a clean 11 to 24px type
ladder at both 375 and 1280.
The home page did have a real defect, just not that one. Its sale and cut
tapes put price, car and a meta line on one row and truncated the meta, with
a comment explaining that truncating kept phones at 375px. It did, but at 375
it cut up to 212 of those pixels off the line, which is more than half: every
row ended mid-word with the yard gone, and the yard is what stops two
identical cars sold in one crawl batch reading as a duplicated row. The meta
takes its own line under the car below sm now and wraps rather than clips,
which is what the /market rows already did. Measured after: clipped text
nodes on the home page at 375 went 72 to 0, worst overflow 212px to none, and
desktop is untouched - the rows are still one line with the same intentional
ellipsis on a long yard name.
Teach the sitemap scanner carma's detail shape
11:01 pm AWST · 162e860
carma's first crawl reported 5 found, 0 extracted, 5 "extractor returned
null" — against a sitemap holding 1,443 cars. The scanner's generic pattern
wants a vehicle-ish word followed by one segment, so /product-detail/367428
matched nothing while /used-cars/peugeot-3008 matched everything: about thirty
browse pages, which is what those five errors were.
That is the failure mode the platform-detail list exists for, and carma is the
fourth entry in it after iMotor, Cox Radius and stock/details.
Worth recording that the probe had this right and the crawl did not. The probe
opens URLs from the sitemap and requires a priced car; it found the 1,443 and
parsed three of them with the existing jsonld extractor. The crawl then
applied a different rule to the same sitemap and kept the wrong thirty. Two
components disagreeing about what a listing URL looks like is a class worth
watching for, not just this instance.
Four more charts get their labels out of the scaled box
10:52 pm AWST · 4c84e83
The model page had the same defect the valuation page did, on all four of its
scatter's caption clipped at the viewBox edge exactly as the other one did.
This is the third page to need the percentage-positioned HTML label layer, so
it stops being copied by hand: components/ChartText.tsx now carries ChartBox,
ChartLabels and ChartText, with the two rules the /value charts learned the
hard way baked in. Line-height is pinned to 1, because overflow-x-auto
computes overflow-y to auto and an overhanging line box turns a chart into a
silent vertical scroller. The layer is aria-hidden, because loose axis numbers
read to a screen reader as noise beside the SVG's own label.
All four converted: the scatter, the Book, the vintage ladder and the range
bar. Verified at 375 and 1280 - zero SVG text nodes in any of them, every
label 11px or 12px, no label overflowing its box, no vertical clipping, and
the interactive marks the readout and the brush depend on all intact: 800
brush marks and 776 sync marks on the scatter, 20 on the ladder, 105 on the
Book.
The scatter's legend also had to stop calling its line a kilometre rate. That
cloud is every live car of the model, all years at once, so the gradient
through it carries the age that comes with kilometres: for the Ranger it
reads 2,302 dollars per 10,000 km where valuation.ts measures the within-year
figure at -862 and refuses to pool the two for precisely this reason. The
legend now states the span it pooled and says the two move in it as one.
Three things carried over from the /value work while the file was open. The
fit spans the odometers observed rather than starting at zero, where it
asserted a price for a car that has never turned a wheel. The x axis takes
its ticks from the scale, since halving kMax produced arbitrary 216k and 432k
labels and hand-rounded thousands collapse on a near-new cohort. Mark size
and opacity follow n, because 800 points at a fixed radius is a white mass
rather than a cloud. And the vintage ladder drops nice() for the reason the
others did: rounding the domain outward on top of the padding pushed the axis
past its own last tick.
Sibling channel: 415 candidates from 88 franchise groups
10:47 pm AWST · f91e0b9
guess-network-siblings reads the pattern off domains that already crawl —
Gibbons runs Kia and Holden, so Gibbons BYD is a targeted guess rather than a
blind one — and found 88 groups running two or more marque sites. 3,460
guesses, DNS-filtered to 415 that exist, recorded as candidates.
Early conversion is 12.5%, against 3% for the wide {stem}cars patterns and
23% for the dealer-name mine. That ordering is the useful part: guesses
anchored on something real beat guesses anchored on a naming convention, and
both beat inventing domains from group names, which the script's own header
records as having run at 35% once and zero twice.
922 -> 927 dealer sources.
The age ladder fills its frame again, with its labels outside the box
10:37 pm AWST · 1d08552
Capping this chart at 720px was the wrong fix and Taj caught it: the frame
keeps its full width, so the cap left the year counts floating in a void
with dead space beyond them. It was a stopgap against the same defect the
scatter had - 10-unit text inflating to 20px in a 1,100px panel - and the
answer was never a narrower box, it was labels that do not live in the box.
So the ladder now splits the way the scatter does: gridlines, boxes and
median ticks stay in the SVG, while the year labels, the n= counts and the
price ticks become HTML at percentage coordinates. It fills the panel at any
width and its type holds 11px and 12px, verified at 375, 820 and 1280 with
zero label overflow, no vertical clipping, and the five data-label hit
targets ChartInteraction reads still intact.
Dropped nice() here too, for the reason the scatter dropped it: rounding the
domain outward on top of the padding pushed this axis to k while its last
tick stayed at k, so the boxes stopped short of the plot edge and read as
a gap before the counts. The widest box now reaches the edge of its range.
The trend says which years it pooled, and refuses to draw a flat one
10:28 pm AWST · e4762f0
The scatter rebuild's plan review returned seven confirmed findings out of
thirty tested. Four were already in the shipped component - the fit bounded
by observed odometers, nice() dropped, the bottom margin at 56 units. These
are the other three, plus one falsehood the review found in the engine's own
copy.
A cohort can span thirty-three model years. Corolla with no year given draws
299 comparables built 1993 to 2026 as one undifferentiated grey, and the
line through that cloud is a gradient of kilometres AND the age that travels
with them: valuation.ts measures the pooled coefficient at 1.6x to 3.7x the
within-year one and names this cost - splitting the two cut median error
13.5% over 46,576 cars. The panel now states the span it pooled, and where
the span is real it says what the extra tilt is, so the number can be
reconciled with the engine's own within-year figure instead of contradicting
it. Single-year cohorts print neither clause, because for them the two fits
are the same measurement and there is nothing to disclaim.
Two reviewers disagreed about the line itself: one to draw the engine's
within-year coefficient, one to keep the pooled fit and name its scope. Kept
pooled. A within-year line over a multi-year cloud is deliberately shallower
than the dots it sits among, and a line that visibly misses the points it is
drawn through is a worse defect than two coefficients that each say which
they are.
robustKmSlope floors a non-negative median at zero, so about one eligible
cohort in sixteen produced slope === 0 - and a horizontal line labelled
"trend" asserts that kilometres cost nothing on a set whose own middle slope
said otherwise. Nothing is drawn there now, and the footnote gives the real
reason rather than the thin-data one.
The x tick labels came from hand-rounded thousands, which on a near-new
cohort reads "0k, 0k, 1k, 1k": distinct positions wearing repeated numbers,
which no dedupe can fix because only the labels collide. They come from the
scale now, like the y axis always did. The aria-label prints exact
kilometres for the same reason - a screen reader should not hear that a 10km
to 12km cohort spans "0k to 0k".
The axis title also needed leading-none: it does not carry .figure's
line-height, and overflow-x-auto computes overflow-y to auto, so its line
box overhanging the box turned the chart into a silent vertical scroller.
Verified at the floor scale, 0.737, where the review measured 255 against
251: scrollHeight now equals clientHeight at 375, 820 and 1280, every label
computes to 11px, and the title sits 9.6px inside the bottom edge.
Last, the engine: "comparable set too small or km spread too narrow to fit a
slope honestly" fires on cohorts of eighteen comparables spanning over
200,000km, because the binding condition is four cars in a single model
year, not the size of the set. It says that now.
A label cannot hold its size inside a chart that stretches
10:16 pm AWST · e3ea13e
The odometer-vs-price scatter's caption rendered at 28px beside a 12.5px
footnote and was sliced off mid-sentence at "yellow = this car at the".
Both symptoms, one cause: the SVG is 420 units wide and renders 1124px,
so every user unit is multiplied by 2.68 - text included, because SVG
font size is in user units and cannot opt out. The caption's text box
ended at 439 units inside a 420-unit viewBox, and SVG clips there.
So the SVG now carries geometry only and every label is HTML positioned
at percentage coordinates. With width:100% and no height the rendered
box is exactly W:H, so a point (x, y) in user units sits at (x/W, y/H)
of the box. Measured after: zero text nodes inside the SVG, every label
a computed 11px at both 1280 and 390, nothing overflowing any edge, and
the page itself never scrolling sideways.
What the chart gained while it was open: an x axis, which it did not
have at all - the odometer range existed only in prose - with an
"odometer · km" title, because a bare "50k" states no unit. Horizontal
gridlines. The Theil-Sen trend the footnote already quoted in words but
never drew. A black ring and a crosshair on the subject, since one
yellow dot among 486 grey ones is not findable. Mark radius and opacity
that follow n: this panel draws cohorts of 56 and of 486, and one fixed
mark cannot serve both.
Three things the build corrected in the plan, each caught by measuring:
padding the domain and then calling nice() widened it twice and squeezed
19k-296k of data across a 0-350k axis, so nice() is gone (round tick
values do not need a round domain); the trend drawn to the axis edge
implied knowledge at 250,000km on the strength of one car, so it spans
only the odometers observed; and a gridline landing within a few units
of the frame drew a stray rule beside it.
The plan's adversarial review confirmed four findings, and one was not
cosmetic at all. The footnote called the pooled fit "$X per 10,000 km",
but valuation.ts documents the pooled coefficient as 1.6x to 3.7x
steeper than the within-year one because it carries age as well - naming
this very model, "Sportage -2,657 against -716" - and the engine adjusts
by the within-year figure, which it prints in the workings as a
kilometre rate that "measures kilometres and not the age that comes with
them". Two different kilometre rates on one page, and drawing the line
would have made the contaminated one the centrepiece. The line stays
pooled, because it is the only line that fits the cloud actually drawn;
the copy now says so and points at the engine's figure by name, guarded
on that adjustment existing and pointing in the right direction, since
the workings render below this panel and only when the within-year fit
succeeded. The review also caught that the bottom margin was sized for
desktop while the labels hold their pixel size down to the 0.74 floor
scale, where the axis title would have spilled into the legend.
The age ladder below gets a width cap so it is not sitting under a fixed
chart with labels twice the size. A full HTML-label pass for it and for
the model page charts, which share the defect at 1.96x and lower, is a
separate follow-up.
Dutton One: 140 vehicles, and why it is not 471
10:06 pm AWST · f35ba98
Raised from 100 after measuring where their resolver actually breaks: 120 and
140 answer (2.72MB), 160 returns "Runtime exited with error: signal: killed".
The search stopped at the first failure rather than bracketing it precisely,
because the failure mode is somebody's Lambda dying.
That is the ceiling for this source and it is short of their ~471 cars. Every
other route on the schema was tried and none of them enumerate:
getRecentVehicle the one that works; no offset, so it is the most recent
N and nothing older
listVehiclesUpdated returns null items for every filter that parses, and
times out on the ones that do not
getAllVehicles takes a `data` JSON string whose shape the site builds in
a lazily-loaded chunk; six plausible shapes all fail on
"Cannot read properties of undefined (reading 'name')"
getSimilarCars a recommendation engine, not an index — asked for BMW X3
it returns a Mercedes GLC, five at a time
yearVCount broken on their side: "Cannot find module 'node-fetch'"
getAllVData returns undefined
So 140 of ~471 is what this reaches honestly. It is the newest stock, which is
the half that moves, and the source records what it is rather than implying
coverage it does not have.
Read Dutton One's stock from the API its own page calls
09:50 pm AWST · 9fb5e6a
duttonone.com.au renders nothing on the server: a 46KB shell, an empty
__NEXT_DATA__, and a sitemap of route templates. There is no HTML to parse
and no path to guess, and it is a multi-showroom prestige group — Caringbah,
Brisbane, Melbourne, Dutton Garage Richmond — which is the end of the market
this corpus covers worst. carma and cars24 are the same shape.
Checked before touching it: robots.txt says Allow: /, there is no
terms-of-use page, and the privacy policy says nothing about automated
access. Nothing here spoofs a browser, solves a challenge, or retries a
block, so SPEC's rule is not in play — it forbids pretending to be something
we are not, not pretending a site is unreachable when it is not.
politeFetch gains an optional JSON POST body and nothing else. The SSRF
guard, the target host's robots.txt, per-host serialisation, the crawl delay
and the honest SpotlotBot agent all still apply; a method and a body are not
a disguise.
The schema's own quirk is what made it look shut. Its resolver calls
.replace() on its arguments, so `limit` must be a String: limit:3 fails
validation, limit:"6" works, and every "Cannot read properties of undefined
(reading 'replace')" was a missing string argument rather than a permission
problem.
Two limits, both deliberate. 100 vehicles a call, because 200 returns
"Runtime exited with error: signal: killed" — their Lambda running out of
memory because we asked for too much, which is worth recording so nobody
finds it again by knocking their site over. And getRecentVehicle takes no
offset, so this is the 100 most recent cars rather than the whole yard: a
sample of the flow, not a census, and it must not be read as one.
The API key is read from the environment and stays out of the repository. It
is their key, shipped to every visitor in their own JavaScript; committing it
would be publishing it on their behalf. Read queries only — the same schema
exposes setEnquiry and setSYCE, which land in somebody's inbox.
100 listings stored through observeListing, so events, vehicle identity and
raw captures behave exactly as they do for an HTML crawl.
Strip the suburb and the department out of generated domains
09:11 pm AWST · 32923b3
174 New South Wales dealers holding 4,908 listings had no candidate domain
tried at all. Reading their names says why: the stem carried things a dealer
puts in its trading name and leaves out of its hostname.
Audi Centre Sydney Pre-Owned is audicentresydney.com.au
Macarthur Nissan - Used is macarthurnissan.com.au
Illawarra Used Cars Wollongong is not illawarrausedcarswollongong.com.au
Two fixes. The suburb is stripped because we already hold it in its own
column — putting it in the hostname is our error, not the dealer's. And a
qualifier list drops the department names: pre-owned, demo, wholesale,
direct, prestige, outlet, clearance, trade.
Both narrow the generated set rather than widening it, which matters after
the wide-pattern pass converted at 3% against the narrow pass's 23%: more
spellings is not the same as better ones.
Place 8,351 stateless listings from their postcode
09:01 pm AWST · b4ff79d
16,717 live listings carried no state — 8% of the corpus — because state is
read off the page and plenty of dealer sites never print it. That is enough
to distort any per-state view, and it was distorting one: the state table
showed Queensland ahead of New South Wales with 16,717 cars unplaced.
Half of them publish a postcode, and Australian postcode ranges map to states
exactly by Australia Post allocation. That is a lookup of a published
standard, not an inference about a car.
Checked before trusting it, against the 172,458 live listings that carry both
a state and a postcode: 170,760 agree, 1,698 disagree, none unmappable —
99.0%. The disagreements are real, mostly a group's head-office postcode
against a branch's state, which is why the published value wins. Derivation
only fills a blank, so that 1% can never move a number that was already
known.
Kept as a separate view rather than backfilled into listing.state: what a
source published and what we worked out are different claims, and collapsing
them loses the ability to tell them apart — the same reasoning that retains
raw captures.
NSW +2,868, VIC +1,720, SA +1,638, QLD +993, TAS +902, WA +230. The gap
between Queensland and New South Wales narrows from 4,118 to 2,243, so part
of Queensland's apparent lead was New South Wales sitting unplaced. 8,366
remain genuinely unknown: no state, no postcode, no suburb.
Do not seed a domain that redirects into a source we already have
08:16 pm AWST · 645dd02
waratahautocentre.com.au redirects to newlambtonautocentre.com.au. The probe
followed the redirect, found real cars, and reported a usable host — every
step true — and seeding it created a second source publishing the same 46
listing URLs as a dealer seeded four days earlier.
The integrity check "no live URL held by more than one source id" caught it,
which is the right check and the wrong moment: by then both sources had
crawled and both had listings. 46 listings merged into their survivors,
dealer:waratahautocentre disabled and pointed at newlambtonautocentre.
politeFetch already reports the final URL after redirects and the probe
already records it as the inventory URL, so the landing host was known before
anything was seeded. The seeder now compares it against known source hosts
and reports the alias instead of adding it. The dealer-name check could not
have caught this one: the site publishes no dealer name at all.
Also seeded sweep 3's finds. 889 -> 902 dealer sources, integrity green.
Note that a hand-run crawl wants --minutes
07:54 pm AWST · f176020
dealer:skymotors held a 159-source crawl for 67 minutes having fetched zero
pages. The host completes its TCP handshake and never answers; politeFetch
bounds each request at 30 seconds, correctly, but a pass has no deadline of
its own unless --minutes is given, so every URL timed out and was retried
until the process was killed.
Two costs, and the second is the one to remember. The pass stalled, which is
merely slow. And it left a crawl_run with finished_at null, which under the
eight-hour dead-run rule would have refused every subsequent crawl — the box's
included — for eight hours. The process closed the row on its way out, so
this time it cost nothing; a SIGKILL would not have.
No code change. politeFetch's 30s per-request bound is right, and the
deadline mechanism already exists in crawl.ts. This is an operating note in
the file most likely to be read by whoever next runs a crawl by hand.
Seed from the candidate table instead of a script per sweep
07:27 pm AWST · 8baead9
Sweeps 1-10b each got a hand-written seed-discovered-N.ts. That was fine when
a sweep found 38 sites and stopped being fine at 159, and it put the evidence
in a log file rather than the database.
probe-candidates already records everything a seed needs in
discovery_candidate.detail — platform, inventory URL, dealer name, three
priced cars — so this reads that, stamps seeded_source_id so a host is never
seeded twice, and sizes maxDetailPages from the link count the probe saw
rather than giving a 500-car yard a suburban lot's allowance.
Seeding is still a decision: nothing moves without --apply, and the dry run
prints the yard, platform and cars for each host.
It carries the duplicate guard the hand-written scripts did not. Of seven
candidates that looked new, six turned out to be second domains for dealers
already crawled — Perth City Subaru against dealer:perthcityautogroup, Kerry
Holden against dealer:kerrysusedcars — and seeding them would have added
copies of cars we already hold. Matched on "ALREADY COVERED by dealer:"
specifically, because records written before today also say "by
market:autotrader", and a marketplace carrying a yard's stock is not coverage
of that yard.
889 dealer sources enabled.
Sweep 10b: 36 dealers a marketplace was standing in for
06:39 pm AWST · 0cdd986
The duplicate check asked whether any source already publishes a candidate's
dealer name. westsideauto.com.au came back "already covered by
market:autotrader — 813 live listings", which was true and the wrong
conclusion. 40 of sweep 10's 45 rejections were that same shape: 32
autotrader, 8 carsguide, and only 5 genuine overlap with a dealer source.
A marketplace copy is not coverage. It is one intermediary's view of a yard
we do not crawl, on a site that already answers 403 to the honest UA on its
index pages, and it can go dark without the dealer touching anything. The
yard's own site is the better record — whole inventory, first-hand prices,
nothing in between. Ross Gray Motor City lists 1,032 vehicles on its own
site and reaches us today only as autotrader's copies of it.
probe-candidates.ts now counts only `dealer:` sources as coverage, so the
check keeps catching real duplicates — the five that were — without
rejecting the dealers a marketplace happens to have found first.
853 -> 888 dealer sources enabled. 706 at the start of the day.
Sweep 10: 159 dealer sites, from names the corpus already held
06:32 pm AWST · da49437
Sweeps 1-9 mined links out of pages already fetched and had thinned to 3.4%
usable — that channel finds whatever a dealer site links to, which is mostly
marketplaces, finance arms and platform vendors. This one starts from the
2,587 dealer names already in the corpus, subtracts the 1,447 a dealer source
crawls directly, and asks only what the remaining 1,140 businesses call their
domain. Nothing is invented: every name belongs to a dealer we can watch
selling a car.
2,148 domains generated, 966 resolved, 907 probed, 208 usable — 23% against
3.4%. 45 of the 208 were dropped as duplicates of dealers already crawled
under another domain; that check is a week old and would otherwise have
seeded 45 sources whose cars we already have.
706 -> 853 dealer sources enabled.
Platform split: 86 jsonld, 65 easycars, 3 easylist, 5 from the backlog drain.
Names and states are sparse because most sites do not publish a dealer name
on the listing; the crawl fills both from the listings. The host stands in
where the name is unknown rather than inventing one.
Mine dealer names into candidate domains, and sweep them in batches
07:37 am AWST · 89b03ac
The corpus knows 2,587 dealer names and crawls 1,447 of them directly. The
1,140 it does not are the discovery channel: every one is an Australian
dealer that is demonstrably trading, because we are looking at a car they
have for sale, usually seen through a marketplace listing rather than their
own site. Their website holds the rest of their stock.
This beats every channel tried before it because it invents nothing. Mining
links out of retained pages surfaces marketplaces and platform vendors and
converts at 3.4%; {suburb}{marque} guesses at dealers that may not exist;
manufacturer locators and trade directories yielded zero between them
(docs/research/dealer-site-directory.md). This starts from a real business
and guesses only what its domain is called — the shape both of yesterday's
successes had, Kerry Holden at kerryholden.com.au and Major Motors RAM at
majormotorsram.com.au.
2,148 domains generated, 966 resolve, 907 recorded after collapsing hosts
two dealers both claim. DNS before HTTP on purpose: a lookup does not touch
the dealer's server, which is what makes it the right filter in front of one
that does.
probe-sweep.sh runs them in batches. probe-candidates.ts fans out with
Promise.all and records only once every host finishes, so one 503 with
retry-after 600 stalled a 98-host batch for the better part of an hour with
nothing written.
Two bugs found by running these rather than reading them:
* The first --record reported "recorded 966" and wrote none. Two dealers
whose names reduce alike put one host in a chunk twice, Postgres
rejected each chunk with "ON CONFLICT DO UPDATE command cannot affect
row a second time", and the summary counted rows attempted rather than
rows written. It now dedupes, counts what it wrote, and exits non-zero
if a chunk failed.
* probe-sweep.sh wrapped each batch in `timeout`, which macOS does not
have. Every batch died with exit 127, the loop continued, and thirty
batches "completed" in nine seconds having probed nothing. Replaced with
a background watchdog, and a non-zero batch now prints its tail instead
of passing silently.
The probe verdict answers the seeding question, not the parsing one
06:54 am AWST · df317c0
Seeding stays a human decision by design — every seed list in this repo was
written by hand after reading a probe. But the verdict said
`jsonld, https://…/stock, 39 links`, which answers "does an extractor
recognise this" and nothing a person can decide with, so deciding meant
opening the site anyway.
It now reports what the yard is. confirmVehicle already opened three URLs and
threw two away on the first success; it keeps all three, and the dealer name,
suburb and state that were sitting unread on the same listing. Three cars
with prices separate a suburban Hyundai lot from a supercar boutique or a
caravan yard at a glance.
And it checks whether we already have them. Australian dealer groups run a
domain per marque — one Kia Seltos here is advertised on thirteen Dutton
domains, same VIN, same $28,990 — so a new host publishing a dealer_name we
already crawl adds copies, not cars. Matched on the name read off the yard's
own listing, because the domain is the thing that differs. The seed-ready
JSON now emits only hosts that are not already covered.
On fourteen real candidates: twelve rejected with reasons, kerryholden.com.au
flagged DUP against dealer:kerrysusedcars, and majormotorsram.com.au seeded
as "Major Motors RAM — 2026 RAM 1500 $139,950 · 2024 RAM 1500 $99,950". The
Kerry Holden line is the one that pays for this: the old verdict would have
read `jsonld, 39 links` and someone would have seeded the duplicate.
Also fixes a bug the run found rather than the reading. run-discover.sh
builds its host list from list-candidates.ts's stdout, and dotenvx prints its
banner there, so under bash the chain has been probing `injected`, `env`,
`(4)`, `tip:` and its own `--record` flag as hostnames every week — a DNS
failure and a discovery_candidate row each, all counted in "36 probed, 0
usable". Filtered in the shell and rejected in the script, because the script
writes the table and cannot trust its caller. 16 junk rows deleted; 141 real
candidates remain, 98 still unprobed.
Both directory channels for dealer discovery are dead ends
06:33 am AWST · ceb9d50
Tested rather than argued, after recommending the manufacturer-locator
channel on a robots.txt check. Ten brands, five trade associations, zero
dealer domains between them.
Manufacturer locators fail two ways. Toyota and Ford answer 403 to the
honest UA on every path including sitemap.xml — Toyota's is a 366KB WAF
challenge. The ones that answer 200 are worse news: Mazda's 237KB
/dealers page and Hyundai's 122KB one contain no dealer hosts at all,
because the locator is a JavaScript map that fetches its list afterwards.
Mazda names /api/dealer-search, which 405s a GET and wants a POST of
unpublished shape. Forty brands of that, for data that may only ever
point at brand.com.au/dealer/x rather than the dealer's own domain.
Association directories fail simply: member lists sit behind search forms
or logins, and every external host on the pages that load belongs to the
association's own estate.
The error worth recording is mine and it is the day's recurring one. I
checked robots.txt, saw permission, and recommended a channel — but
robots.txt answers "are we allowed", not "can we fetch", and Toyota's WAF
answers the second question differently. One signal read as evidence for
another, exactly as in DATA-QUALITY 35 and 37. The test cost fifteen
minutes and was available before the recommendation, not only after it.
Nothing shipped. docs/research/dealer-site-directory.md carries the table
so this is not rediscovered.