The three collapsed imotor sites needed a sitemap, not an ingest
07:46 pm AWST · 339d88d
Fixed in-session rather than as a separate build. The imotor Gatsby+Algolia
theme that hollowed out buckbyram, patrickautotraders and cheapcarstas (42)
looked like it needed an Algolia ingest — every HTML route serves the same
12 cars and the yard sits behind the site's own dealer_stock_<id> index. It
did not: imotor publishes sitemap-stock.xml on each site (242/81/43 car
URLs, each matching its own nbHits), full inventory as ordinary detail URLs
the existing extractor parses, simply unlinked from sitemap-index.xml.
The dealer-studio config fix (41), not new code: indexUrl moved to the
stock sitemap, old /stock kept as a secondary index, collapseAcknowledged
cleared because these are fixed now, not deferred. Verified end to end —
a detail page parses clean (2024 GLA250, $52,990, 22,974km, VIN), and a
bounded crawl re-observed 188 cars and climbing against the collapsed 12.
Also checked the blast radius: 114 of the 115 imotor sites already pointed
at sitemap-index.xml are healthy, so their index links their stock sitemap.
The three that broke were the ones still pointed at the HTML /stock page —
exactly the surface the new theme emptied. The spawned Algolia-ingest task
is withdrawn; DATA-QUALITY 43 records why the sitemap beat the API.
A starved pass is excused until it has had its chance
07:28 pm AWST · 6ec7631
The heartbeat excuse only covered a longer pass in flight, and fired "the
box may be down" eleven minutes after the deep chain finished cleanly: the
fast beat was eight hours old because deep had held the crawl slot for
eight hours, and fast's next timer slot was still two hours away. Now the
excuse extends 1.25 of the starved pass's own periods past the moment the
slot frees — fast gets until the first slot it could actually catch, and
alerts only if it misses that too.
Caught by running health after the first clean chain in five days, which is
also worth recording: integrity_run 2026-08-23 11:25:35, scheduled,
failures=0, 21 checks — the statement-timeout fix holding on the box
itself, and the 136-hour alert gone.
A known break is acknowledged, not silenced
07:10 pm AWST · 2eb2d6b
The three imotor sites that collapsed on the vendor's Algolia theme cannot
be fixed with a config row — every HTML route serves the same 12 cars, and
enumerating the yard means reading their own search backend the way Dutton
One's ingest reads AppSync. Until that ingest exists, check_found_collapse
would stay red about a break we already know, and a permanently red alarm
is a wolf-crier: the next NEW collapse would land in an inbox already
ignoring the email.
So crawl_config.collapseAcknowledged = 'YYYY-MM-DD' suppresses a source
from the check — for 14 days. Acknowledgment has a shelf life: a break
still unfixed a fortnight later has earned its way back into the alarm.
The three sites carry the flag with the diagnosis in robots_notes; their
crawls continue reading what is readable and the delist check keeps
probing the rest.
The client noticed before we did; now the system notices first
07:06 pm AWST · d2f742f
Kiel checks Westside's yard page daily and watched it not change for five
days, because the crawl behind it was blind (DATA-QUALITY 39) and nothing
watched a single source against its own history. Two changes.
check_found_collapse (0198), wired into health: a source whose best found
over the last two days is under a quarter of its trailing week's best,
floor 40, alarms — and no recent runs at all counts as collapsed to zero.
Run today it immediately caught three breaks the manual audit missed:
buckbyram, patrickautotraders and cheapcarstas (imotor's new theme is
Algolia-backed client-side search — needs an extractor of its own, parked
and now watched), and tonylahood (easycars moved its index to
/used-cars-in-croydon/ — fixed with one config row, extractor verified on
the moved page: 30 listings, working pagenum links).
crawl_config.priority: a dealer with a portal account gets fast passes that
do the deep pass's job — full index traversal, a 40-page detail budget the
existing rotation spends on never-seen cars first, and complete left honest
so the delist check may run. Delisting safety is unmoved: it rests on
presentUrls from the traversal, the same invariant --max-detail documents,
not on how many details were fetched. Westside is flagged.
Verified live on a fast pass against cowramotors (flagged for the test,
unflagged after): found=46 against the collapsed 21, one arrival caught,
one price change, complete=true, delist check ran and correctly refused a
still-200 candidate.
Exits now reach a watched dealer's page within a couple of hours instead of
within twelve — and the next platform-vendor rollout that blinds a cohort
of sources emails us the day it happens instead of surfacing as client
feedback five days later.
Freshness reads the listing, and the audit's last two finds
06:37 pm AWST · 03e950a
check_price_freshness outgrew its query shape: it scanned listing_current —
per-row latest-observation lateral, vehicle join, price-change count — to
answer a question about two columns. 112 seconds against base tables at
224k live listings (measured); the view version exceeds 120s outright, so
integrity check 17 died nightly. It now reads listing.last_seen_at via
listing_status_idx, verified within one hour of the true latest observation
on 500 of 500 sampled live listings, against thresholds of 24h/48h/14d.
One stated semantic change: the population is every live listing, priced or
not — a 2.6% widening. Migration 0197; the suite passes end to end again.
The stale-listing warning, decomposed rather than ignored: 15,882 total.
10,700 are marketplace window-churn (autotrader's 1,500-page cap never
revisits its tail — structural, already documented). 3,716 were real:
nineteen dealer-studio sites collapsed to exactly one index page (21 cars)
on exactly Aug 16 — the vendor rolled a client-side-pagination theme to a
cohort of its customers, the same failure Westside had, at platform scale.
No extractor commit landed in the window; it was their change. All 19 now
read sitemap-custom.xml, individually verified first (3,906 car URLs).
DATA-QUALITY 41 records the signature for when the vendor rolls the theme
to the remaining 446.
Also caught by the suite once it could run: two vehicles created Aug 21
with null make — the dealer group's own DMS published the cars as
"unknown-unknown" — identified from the group's other URLs plus VIN
prefixes (Nissan Qashqai J11, Mazda CX-30 DM) and queued for taxonomy
review rather than left as unexplained out-of-scope rows.
The five-day audit: a leash, a redesign, and a wolf
06:10 pm AWST · af963ec
Three faults found auditing five unattended days, none of them the crawl.
The leash: every deep chain tail since Aug 18 died on "canceling statement
due to statement timeout" — compute-insights mid-corpus, check-integrity on
the delisting check — so no scheduled integrity row was filed for five days
and health rightly alerted. Neither query is slow: the delisting check runs
in 955ms warm, the corpus pages at ~1.4s. They died because service_role had
no statement_timeout of its own and inherited the authenticator's 8s — the
pipeline ran on the same leash as an anonymous page view while the corpus
doubled. Migration 0196 gives the service key 120s; anon and authenticated
keep their short ones. pageByKey now also retries a page that hits a
timeout, halving the page size each attempt — the corpus read paged for five
minutes and lost everything to one bad page. Only timeouts retry; every
other error still throws on sight. DATA-QUALITY 40.
The redesign: Westside Auto Wholesale went 1,795 listings found to exactly
18, overnight on Aug 17/18, complete=true every pass. Their site moved
pagination client-side — /cars?page=2 renders the same 18 cars as page 1 —
so 1,905 listings sat live with ageing prices while the delist check
correctly refused to call 200-answering pages sold. Their robots publishes
sitemaps/individual-cars.xml with 1,803 car URLs; indexUrl moved there, the
carma pattern. Config row, not code. Verified flowing again: 82 cars
re-observed in the first five minutes. DATA-QUALITY 39.
The wolf: a deep pass at 990 sources runs about eight hours, and while it
holds the crawl slot the 2-hourly fast heartbeat rightly goes stale — which
the new heartbeat alarm would have called a dead box twice a day. Staleness
on a short-cadence beat is now excused while a longer pass is legitimately
in flight, and an unfinished pass older than 1.25x its own period alerts as
wedged. An alarm that cries wolf twice a day is an alarm nobody reads.
Verified before committing: compute-insights read all 200,000 listings and
wrote 5 insights while the box's deep crawl had the database busy — the
exact condition it died under for five nights.