The fast pass steps around the deep pass instead of skipping its slot
11:27 pm AWST · 18889e7
Deep passes grew to eight hours twice a day, and the global in-flight
guard turned that arithmetic into starvation: 62 fast slots skipped
against 28 run in one week, a price cadence of twice a day pretending to
be twelve. The risk the guard exists for was never global — it is per
HOST (a host hit by two crawlers answers 503 to both, DATA-QUALITY 18)
— and the open crawl_run rows already say exactly which sources are
being worked.
So the guard splits by what each pass costs to skip:
- A FAST pass reads the open claims and steps around the sources — and
hosts — they cover, crawling everything else. Checked at queue build
and re-checked at dispatch, because a deep pass opens new claims for
hours. Same-host siblings are excluded together, mapped across ALL
sources so a claim keeps holding its host even if its source is
disabled mid-run.
- The claim itself is settled by INSERT order, not by looking: a pass
inserts its run row first and then checks for an older open claim on
the host; the younger claimant closes its row and yields, ties break
on id, so a race the checks cannot see still ends with exactly one
pass on the host. Adversarially reviewed: 26 agents, 14 confirmed
findings, the check-then-insert race chief among them — both passes
sort by the same staleness key, so the fast pass's first pops are
precisely the deep pass's next pops, and "unlikely" was doing a lot
of work the tiebreak now does instead.
- Two FAST passes never overlap: the run row carries an in-flight pass
marker (cleared on close), and a fast-shaped claim stands the next
fast slot down whole, from any machine — restoring the cross-machine
exclusion the global guard used to provide.
- A DEEP pass keeps the global wait: skipping one costs twelve hours of
delistings. Its dispatch now also waits out a briefly-held source
rather than colliding with it, and a deep slot given up after the two
hour wait records a heartbeat and exits red instead of vanishing
(DATA-QUALITY 38).
- health.ts stops excusing a stale fast beat while deep is in flight —
that excuse described the starvation this change removes, and keeping
it would have muted the alarm for the exact regression to watch for.
Verified live with synthetic claims: a held source yields while a free
one crawls; a fast-shaped claim stands the slot down; an all-held
hand-run exits 1 (counted against matched sources, so a typo cannot
launder it to green); the marker writes at insert and clears at close.
A week of red mail traced to four defects, each fixed at its own layer
10:50 pm AWST · 8f2bebf
The deep chain has ended red on 26 of its last 28 runs, and the health
email fired about it again today. The audit (16 agents over the box, the
crawl data, the integrity suite, and the site) found the crawling itself
healthy the whole time — volume flat at ~390k observations/day, all 12
cron jobs green — and four real defects keeping the alarm stuck on:
- The dealer-studio identity guard compared identities as raw strings,
so "q5-72030" against stocknum "72030" read as someone else's car and
~440 valid brisbanecars pages per pass were refused for three weeks —
a fifth of the source's budget spent proving nothing, and the direct
cause of the fortnight-unconfirmed residue. The guard now accepts a
doc that agrees with the page's slug on ANY identity it carries,
including the numeric tail, and still refuses a doc that agrees on
none. Verified against today's retained captures: 12 of 12 pages the
old guard nulled now parse with the right id and price.
- The four-hour-per-source rule the deep unit file has claimed since the
box was built never existed in code. osborneparkkia's 2:30am
maintenance window held a run open seven hours, twice, and both times
systemd killed the WHOLE pass at ten hours — costing the overnight
integrity run and every delisting inference. crawlSource now carries a
real per-source deadline; expiry ends the source complete=false, which
the honesty invariant already makes safe.
- Integrity check 17 hard-failed on ANY live listing unseen 14 days,
but the lapse job exempts dealer listings by design, so a small dealer
residue is permanent and the alarm stopped meaning anything. 0216
splits freshness by source kind: marketplace staleness (which the
six-hourly lapse should hold at zero) stays a hard failure, dealer
stragglers are a named warning with a stall ceiling.
- Check 4c manufactured today's new failure: ingest excluded Penrith's
"Unknown Unknown" placeholder pages on the RAW make, then stored make
as NULL — so the stored row could no longer justify its own exclusion.
Ignored rows now keep the word the rule matched, and the two existing
rows got their evidence back in 0216.
And the exit semantics that let real breaks hide inside the wart: the
suite now exits 2 when checks ran and filed a failing verdict (health.ts
alarms on the recorded run) and 1 only when it could not run — so a red
deep unit means the chain broke, in all three chain runners.
Suite run against the live corpus after these fixes: all checks passed,
3 warnings, exit 0 — the first green verdict since 25 August.