Every change, in the words it was made with

The crawler reads past the row cap it did not know it had

08:13 pm AWST · e79f46e

Two more health alerts, two more causes found, one of them the best bug of the week. PostgREST clamps every response at 1,000 rows, requested range notwithstanding, and on 30 August the enabled corpus crossed exactly that line — so every pass since has quietly dropped two arbitrary sources from its queue, and the rotation RPC lost 44 sources of history, which is where "44 never crawled" came from on a corpus where everything had been crawled. patrickautotraders drew the short straw four passes running; its "index gone dark" alert was this truncation wearing a disguise, on a yard whose sitemap serves 81 cars perfectly well. DATA-QUALITY 16, third appearance: the cap never announces itself.

The three unbounded reads in the crawl path — the source queue, the rotation history, the host map — now paginate in stated order, and 0220 gives the rotation RPC the order its reader was already assuming. The per-source and per-batch reads were already paged and chunked by the people this bit before.

And 0219: the marketplace freshness check hard-failed at 14 days, the same threshold the six-hourly lapse job lapses at — a check and a janitor with one deadline will always meet, and they did this morning, on 77 autotrader listings aged 14.0 to 14.2 days, half an hour before the lapse run that would have taken them. The check now fails at 15

A board refreshed on time is not stale

11:59 am AWST · 5d616dd

The twice-daily boards carried max_age_hours = 12 against a 12-hour refresh clock — zero slack — and the integrity check's time drifts with deep-pass length. Last night's longer pass checked at 00:39, found every 12:25 board 12.23 hours old, and went red over fourteen minutes of clock arithmetic. Fifteen hours keeps the stall alarm (a stopped refresher blows past it by the next check) without alarming on the schedule doing exactly what it says.

A fresh delisting gets time for its paperwork

07:09 am AWST · 6a44e4c

The partial-runs check counted a delisting as unjustified the moment it landed, though every delisting is already page-confirmed and the next deep pass files the corroborating traversal as a matter of course. Each deep-pass death therefore produced a few hours of 'unjustified' delistings, a health email about them, and then a silent healing — the recurring failure Taj kept being mailed about, including 22:59 last night, whose two flagged events were both justified again by midnight. Fifteen hours is one deep period plus margin: paperwork still missing after that has actually been missed.