Method · Changelog · 14 Aug
This is the repository’s own history, newest first, each entry the message its change was committed with and nothing added after the fact. The messages say what changed, why, and what they found wrong on the way, including in earlier entries. A figure quoted in one is the figure on the day it was written. A day to a page.
746 changes over 22 days since Monday 10 August 2026 · as held in the repository at 10 Sept 2026, 09:03 pm AWST, when this build was deployed. The defects that reached the database have their own register on the data-quality page.
10:35 pm AWST · 0e00635
A 1994 TVR reached the review queue with model "Other". The make is real — British sports cars, out of production since 2006 — and the feed simply had nothing useful to say about the model, which is not a reason to drop the car.
Added to CANONICAL_MAKES and mapped. Queue back to zero.
10:16 pm AWST · a2fd4c2
Seven broken findings from the six-section audit, all fixed but the one that belongs to the crawl track:
Every grid child on the site now carries min-width:0. Grid items default to min-width:auto, which lets a wide table or chart propagate its intrinsic width THROUGH its overflow-x-auto wrapper: the wrapper never scrolls and the whole viewport pans instead. Measured at 375px, /market was 643px wide, /coverage 803px, home 390px. One zero-specificity rule in globals.css restores the scroll boundary to the wrapper, and the two flex rows that still forced width (shrink-0 right-hand spans in the two-price and demo panels) wrap onto their own line on phones. All three pages now measure exactly 375px at phone width.
/value stops shipping the whole catalogue. The full pick-list serialised to 3.9MB and rode into the page as a client prop, 632KB gzipped on the wire before the picker could hydrate, on the site's primary tool. The page now ships the makes list plus the slice for the URL's own subject (so shared links hydrate complete), and the picker fetches per-make and per-model slices from /value/options, served off the same server cache the page reads. The page is now ~48KB raw before compression, a thirteenfold wire reduction, and picker counts gained their thousands separators on the way through. A failed slice fetch says so in the placeholder rather than presenting an empty list as a fact about the corpus.
km=abc no longer values a phantom. Strip-then-Number turned any digit-free string into 0, and the car was valued as a zero-kilometre example while the hero hid the 0 as falsy. A string with no digit now refuses with 'must be a number', and a genuine 0 km prints, because a valuation whose subject is invisible is unfalsifiable.
The market state board knows the eight states: anything else folds into the honest UNKNOWN bucket at build time, so a bad feed can never mint a ninth state again. The board was rebuilt; PAKENHAM is gone from it.
The /api sample JSON was black-on-near-black inside the yellow panel, 1.1:1 contrast on the page's flagship honesty block. The Code component states its colour instead of inheriting the panel's ink.
Autotrader AU's extractor regression (zero yield since 13 Aug after 8,875/1,878 the day before) is real and is the crawl track's to fix; a task chip carries the full diagnosis. The coverage page was truthfully reporting a dead crawl and needed no change.
10:02 pm AWST · 6dadda3
Front-to-back review feedback, each item run to its root:
Pakenham is a suburb. Pakenham Mazda prints it where its platform keeps a state, the listing-stated path trusted extractors where the dealer-location fallback never did, and 154 listings put a ninth state of Australia on the market board. The ingest gate now normalises the listing-stated state through the same eight-code rule as the fallback, the rows are repaired to exactly what guarded code would have written (VIC, dealer_location provenance, migration 0133), and the state panel renders the null bucket as 'not stated', last, instead of as UNKNOWN among the states. The panel also drops anything else non-state as a belt.
The Time and Sales tape was the third child of a two-column grid, wrapping alone into a half-width orphan beside empty space. It spans the full row now, with the tape itself in two internal columns so forty events read as a block rather than a tower.
Coverage grew past 700 sources and outgrew scrolling: a search box and state filter, in the house idiom, with the server rendering every row and the client only toggling hidden. 'mazda' filters 731 to 39 instantly, plus WA to 6, and the count line always states n of total.
The on-roads panel comes off the market page. The measurement is real and the 300-pair bar is honest, but three qualifying states read as a coverage gap, not as rigour. The board keeps computing and the panel can return when five states clear the bar.
The rates strip moves from /market to /reports: a reader of live stock does not need the cash rate, and a reader of a weekly report is exactly who wants the backdrop.
The insight was never frozen. It is retested after every crawl, four hypotheses each time, and the colour premium keeps being the only one to clear the bar; it had moved 6 to 7 percent during the very days it looked static. The panel now says so: retested N hours ago, held through 15 recomputes since 12 August, the only one of 4 tested to pass. Stability now reads as re-earned rather than stale.
09:34 pm AWST · 97284f9
The reports mechanism: market_report holds immutable snapshots of closed UTC periods, written by cron the morning a period ends (weekly Mondays 00:15, monthly on the 1st at 00:25) and never rewritten, the same way an observation is never edited. /reports renders the archive plus the running week, measured live by the same generate_market_report_data the cron uses, so the preview and the eventual report cannot disagree. The print stylesheet is the PDF export; the share paragraph is the report in one block for wherever Taj is posting; the first weekly report writes itself on 2026-08-17.
It took three drafts of the measurement to deserve the page, and the drafts are the story:
v1 measured the crawler and called it the market. 189,308 "new to market" in a 190,000-listing corpus was discovery, not listing; a live delta of +127,150 was coverage; the ask movers were composition (BMW X3 "up 39.9 percent" on an n that went 93 to 413). Caught by reading the output against the corpus before anyone else saw it.
v2 removed those, framed corpus growth as corpus growth, replaced movers with repricing counts, and aligned valuations with the site counter's exact filter, proven by both reading 545. The adversarial review then found what v2 left standing, and it was blocking: sales counted delisted listings, and dealer-group syndication exits one sold car as up to thirteen listings. 5,895 "confirmed sales" were 3,719 physical cars; the repricing table had the same disease under a caption claiming coverage could not move it, which the corpus tripling inside the window falsified.
v3 counts cars everywhere. A car is confirmed sold in the week of its final exit, none of its listings still live, its value the final ask of its last listing, once. A repricing is one car cut on one day, however many syndicated copies repeat it; the dedup landed on 7,577 for the window, the reviewer's independently computed figure to the digit. The basis strings ride in the payload and the page renders them from data, so the copy cannot drift from what the SQL measured. The share paragraph carries its units ("cars confirmed sold", "in final asking prices", "once per car per day") because it is the one artefact designed to leave the page without its footnotes, and an archived report's corpus figure says "at close" rather than "now".
Also from the review: window bounds are pinned to UTC rather than the calling session's timezone, so a manual run from Perth cannot write a shifted week into an immutable row; a failed archive read renders as a failure instead of the factual claim "no closed period yet"; the publish-date sentence is derived so it cannot go stale; and the week-so-far cache key carries its Monday, so the first hour of a new week cannot serve the closed week as the running one.
One reviewer claim was dismissed with evidence: the valuations filter allegedly diverged from the deployed counter, but the live function carries the api exemption and both figures read 545.
Withheld, with reasons in the payload's absent array: new-to-market and median-ask movement, until coverage stabilises enough that a week's n holds still. The writer is proven idempotent and refuses open periods; the pre-monitoring test row was removed rather than published, because a report of zeros for a week nobody watched is true and misleading at once.
Migrations 0129 to 0132, byte-identical to the ledger, tree in sync at 132 = 132. Integrity 13 of 13.
08:59 pm AWST · 389a6c2
The plan grants brushing exactly once: on the odometer axis, to filter the comparables table, never touching the fit. Both halves land together, because the exception without its table would have nothing to filter and no reason to exist.
The comparables table lists the same cars the cloud plots, one row per physical car, each linking to its listing, sorted lowest-odometer first and saying so rather than posing as representative. It is served in full and default-trimmed by the hidden attribute, so the brush works entirely by toggling rows the server already sent: no request, no invented data.
Drag across the cloud and the table filters to the window. The status line restates on every brush that the chart, its fit and its rung still use all cars, because a filtered table beside an unfiltered fit is exactly where a reader would assume otherwise. A sub-8px drag is a click and clears; Escape clears brush and readout both; the aria path is the table itself, which is complete without the pointer.
The client keeps its ignorance: the brush selects marks by screen position and reads the window's km bounds off data attributes the server wrote. Selection is mark containment, not axis arithmetic. setPointerCapture failure is swallowed rather than allowed to abort the gesture, found by driving the brush with a deliberately bogus pointer.
Cost, stated because the admission was a measured number: ChartInteraction was admitted at 1.3kB for readout and focus, landed at 1,442 bytes, and now carries readout, focus, sync and brush at 3,114 bytes gzipped, against the 107.6kB the rejected library wanted.
The chart set is built. What remains is time: price_over_time earns its ribbon as the rollup accumulates days, and Phase 4 unlocks by SQL.
08:50 pm AWST · 5a56023
Chart set #8, and the smallest of them: the RBA cash rate under the market panels, 432 monthly points held since the table existed and read by nothing until now.
Drawn as steps, captioned as averages. The series is the monthly average of the cash rate target, so each month's value holds flat across the month it describes and no diagonal invents a glide the rate never made. Readout rides the same ChartInteraction as every other chart, one mark per month.
Copy is locked to context, not causation: the strip states the rate and claims nothing about its effect on cars, which no chart here has earned.
It wears no evidence rung, deliberately, and says so on the page. The ladder measures what our corpus supports, and the RBA's series is not thinner because our crawler had a slow week. What official data owes is stated provenance instead: source table, licence, and the month it runs to, all carried on the strip.
No migration: this is a reader, and the context job has been maintaining the table all along. d3 stays server-side (client chunks verified free of it), integrity 13 of 13.
08:43 pm AWST · c50266c
Chart set #4 ships: used asks per model year on one shared, labelled axis, one price per physical car, the same basis as the range bar and the book so the three never disagree about what an ask is. The copy holds the line the chart set drew when it named this chart: a cross-section of today, not a price history. The depreciation claim belongs to model.price_over_time and has to be earned there.
The ladder joins the evidence spine as model.by_year, requirements mirroring model.book because both are the same claim about where asks sit, one sliced by price and one by year. The rung gates the form exactly as chart_rung declares it: rung 1 draws a median tick per qualifying year, rung 2 earns the middle-half boxes. Within a rung the house rules ride per row: a box needs 8 used cars, a tick needs 3, and a year under 3 is absent here and listed in the years table. Live at both rungs: Ranger draws 15 boxes and 3 ticks at rung 2; FJ Cruiser draws 6 ticks and no boxes at rung 1, its strip reading "needs priced cars 30, this cohort has 24". Readout and range-sync reuse ChartInteraction unchanged, the client component is byte-identical, and the active year highlights on year routes.
model_chart_board's years rows carry the new per-year quartiles, vehicle-deduped, validated exactly against live_snapshot before anything read them: Ranger 2021 agrees on all four figures at n=347.
The adversarial review of the diff found three real defects, each confirmed against live data before fixing:
- The axis took its domain from quartiles it never draws. Tick-only years carry p25/p75 in the payload, and the domain used them while the render did not: 172 live cohorts had inflated axes, Cupra Formentor at 80 percent empty chart. One predicate now decides both what draws and what scales; Formentor's drawn span went from 17 to 61 percent. - The hand-rolled ticks reintroduced the exact zero-spread collapse CHARTS-PLAN cites as the reason d3-scale was admitted, and the helper had never actually been installed. Volkswagen Jetta rendered one gridline labelled $10k over a $1,135 window. d3-scale is installed now, server-only as admitted (client chunks verified free of it, and ChartInteraction still 1,442 bytes); Jetta reads seven distinct ticks at $200 precision. The scatter's own y-axis, the original site of the bug, gets the same treatment, and the range bar's min and max labels are keyed by anchor so a one-price cohort cannot collide keys. - The refusal branch conflated unmeasured with insufficient: on a null rung it printed "not enough priced used cars", a fabricated thinness claim whenever the ladder read fails, under a header that may show thousands of cars. Four states are told apart now, and the genuinely thin branch states the measured count, because a thin corpus is reported with its real number, never with an adjective.
Migrations 0127 and 0128 are byte-identical to the ledger; the tree remains in sync at 128 = 128. Integrity 13 of 13 before and after.
08:14 pm AWST · 65c373e
The charts gain their one client component. ChartInteraction carries the two client primitives the plan admits, readout and focus, and stops because our reads are a fetch schedule and a smooth transition between them would be an invention.
Every scale stays on the server. Marks carry their full label and, where sync applies, a precomputed percentage; the client does nearest-mark hit testing in screen space, positions a crosshair and a readout box, and reads labels into an aria-live region for arrow-key traversal. That division is what keeps it at 1,442 bytes gzipped against the 1.3kB the Phase 0 evaluation admitted, and it is why the component never learns what an axis means.
The readout restates n every time. A dot on the cloud reads "2021 used · $45,990 · 63,247 km · one car", because "$45,990" alone is exactly the unearned precision the tooltip rule bans. Book buckets read their range and count, and the shelf buckets say they are the shelf.
The synchronised range panel: pointing at a used car on the cloud marks where it sits in today's range bar. The price-to-percentage maths happened on the server against the bar's own domain; demo cars carry no sync, because placing a demo ask on a used-only scale is the overlay rule's cross-basis mistake in miniature.
The facet invariant is the substance of the change. A year chip thins the data, and the drawn rung is now min(the cohort's ladder ceiling, what the slice's own evidence supports), resolved by first failure against chart_rung_requirement, server-side, in the same render, so no paint exists where a thin slice wears the whole model's rung. Proven in both directions on the same cohort: hyundai/kona draws its scatter at rung 2, and /2026, with 177 cars spanning only 11,014 km, draws the same chart at rung 1 with the strip reading "next rung needs km spread (km) 20,000, the 2026 slice has 11,014". Near-new years are exactly where a depreciation fit lies, and now the frame says so before the fit rule even declines.
Found while proving it: a year slice filtered from the 800-mark cap is not a count of that year's market, and the empty-state copy claimed it was. Old Ranger years read "only 0 cars with a price and an odometer" when the corpus holds them and the cap dropped them. The line now states the cap when it binds. The book and aged panels also say "all years" on year routes, since they are not year-faceted and every chart beside them just filtered.
Verified in the browser end to end: readout, crosshair and sync fire on pointer; focus announces "800 marks", ArrowRight and End traverse, and Escape clears including the sync marker; integrity 13 of 13; tsc and build clean.
Deferred, stated rather than implied: the Vintage Ladder until model_chart_board carries per-year percentiles, the rates strip until econ_series has a reader, and brushing until a comparables table exists, since the plan admits brushing only to filter one.
07:53 pm AWST · 08d94ed
The first crawl run on the reconciled tree failed integrity honestly: oldest ladder measurement 13.0 hours old. 27 cohort keys had been renamed out of the taxonomy by spelling merges (Honda|CR V and Honda|CRV both, twelve Mercedes|* Class variants) and refresh_evidence_series only ever upserted what the live corpus showed it, so their evidence froze at the 2026-08-13 bootstrap stamp forever.
Dead keys are reaped from evidence and ladder_state after the rollup-derived insert, which would otherwise resurrect them from history each cycle; ladder_event keeps its append-only past. Taxonomy keys with nothing live get their live-derived measures written as zero with a fresh stamp, because nothing live is a measurement, not a skipped one, and it is exactly what immediate demotion needs. check_ladder_spec returns clean, integrity 13 of 13, tree still in sync at 126 = 126.
07:47 pm AWST · 8134ba9
Fixed the cars4us badge at ingest, backfilled the history, and two hours later the running crawl pass re-reached that source with the previous build and wrote 30 junk badges back — 378 distinct badges became 408. Nothing alerted. It was found by going to look.
Recorded in entry 20 because the trap is general and not about badges: a value fixed at ingest plus a backfill is half a repair while a long-running process still holds the old code, and the half that is missing gets silently reverted for as long as that process runs.
07:43 pm AWST · 6d35826
Three sessions applied migrations to the same database in parallel, and files are numbered by the order the ledger applied them, so each session shifted the others' numbering. The drift check counted 40 differences and failed the last two scheduled crawls at their final step, which is the trap this project keeps naming: a workflow red for a known reason is a workflow nobody believes when integrity genuinely fails.
Reconciled per the runbook now in OPERATIONS.md:
- Two files on disk had slugs the ledger had never seen at all: raw_capture_pending_drop_idx and raw_capture_pending_upload_idx, both built with CREATE INDEX CONCURRENTLY, which cannot pass through apply_migration. Both indexes exist in the catalogue, so each file's exact content was applied as a recording migration first: a no-op against the live database that gives the ledger its missing rows. Their content survives byte-identical at 0124 and 0125. - sync-migrations --write then emitted all 125 ledger entries at their true numbers: 24 new files, including the six unfiled migrations from the crawl and storage session (unmatched_marketplace_dealers, cohort_median, raw_capture_bodies_move_to_object_storage, source_max_found_with_parse_rate, cohort_listings, source_max_found_with_fetch_rate), which close the runtime landmine where a rebuilt database served a three-column source_max_found to a script reading four. - The 18 stale old-numbered files are removed in this same commit, because --write exits 0 while they remain and the check would stay red forever otherwise. Eight of them carried richer commentary than the ledger recorded; that prose lives on in git history at the old paths, and the check is right that the tree should say what the database actually saw.
The check now prints "in sync: the tree rebuilds the database exactly" and exits 0. Replay order is also correct end to end: the live_snapshot migration creates three indexes at position 103 and the recorded drop removes them at 114, matching the catalogue, which holds only the primary key.
125 applied, 125 files, integrity 13 of 13.
07:40 pm AWST · a75ce3b
07:40 pm AWST · 242c3b2
Ford Ranger could not open. model_ticker assembled the whole position at request time against the live corpus, that outgrew the API's 8 second ceiling at the three largest models, and a page that has never rendered once has nothing to serve stale, so the cache could not save it either. The single most valuable page opened with an apology exactly where the most people would see it.
model_chart_board holds the assembled JSON per slug pair and model_event_board holds the tape as rows, both rebuilt in the six-hourly cycle from the same live_snapshot as every other board, so the whole page agrees about when "now" was. The request path is a primary key read: 0.155ms for Ranger. Ranger, X-Trail and Tucson all serve in under a second. model_ticker has no callers left.
The scatter stops lying about what a dot is. It plotted listings under a caption that said cars, and used syndication runs 1.80x, so a car on many sites was many dots. One dot per physical car now, its most recently observed listing, and points_total counts cars: CX-30's 537 listings became 263 cars. The header's used count is relabelled listings for the same reason, that one the other way around.
The Book and The Shelf ship, first of the chart set. Fixed $1,000 buckets, one count per car, the same buckets the daily rollup captures, so the cross-section and the eventual price_over_time ribbon are one measurement at two timescales. The shelf shares are stated on the chart: Ranger asks sit 16.8 percent in the thousand below a ten-thousand boundary against 4.8 above. Copy locked to "where dealers price, not where cars sell" until exit evidence clears rung 5.
The daily rollup starts capturing that histogram today, per model, because the 12 August hole proved the rule: a day the rollup did not capture cannot be recovered, and the shape of a distribution cannot be rebuilt from its median. 2026-08-13 and earlier are honestly null. The first cut of this died on max(jsonb), which does not exist; plpgsql only parses SQL at first execution, so running it is what found it.
Aged stock is split by condition on the page, because 75 percent of aged stock corpus-wide is new or demo and the "nobody wants this" reading only holds for used. Conditions under 30 dated listings are stated, not drawn.
The rung becomes visible. Each chart sits in a frame keyed off its ladder_state rung, pure CSS: dashed while evidence is thin, solid once it holds, yellow at the top rungs, and a missing rung draws the rung 0 frame because absence of evidence about the evidence is the thinnest state there is. Under the frame, the static evidence strip states the rung and what the cohort lacks for the next one, read straight from the ladder's block columns. Interaction is Phase 3; honesty did not wait for it.
Migration files 0112 to 0116 are byte-identical to what the ledger recorded, the discipline the sync audit taught: 0115 carries an ugly create-then-drop scaffolding statement because that is genuinely what was applied, and a prettier file would be another lie for the drift check to find. Verified: all five match the ledger by content, board counts match the retired RPC on every scalar for CX-30, both new boards are in board_spec, and the integrity suite passes 13 of 13 before and after.
07:40 pm AWST · 918a449
Two vehicles had been sitting in review all day, and they wanted opposite answers.
**Barker Trailers 2014** is a trailer builder. Added to NON_CAR_MAKES as "barker trailers" — matched on make and model together, because the make alone is a surname and could be a dealership's own name. Marking it ignored by hand would have failed the integrity check that every out-of-scope vehicle is one the rules exclude, which is exactly what that check is for.
**Rambler American 440, 1967, 4d sedan** is a car. Rambler is a real marque, defunct rather than unrecognised, and a 1967 sedan at a dealer is a classic somebody is selling. Mapped.
The queue is meant to mean "a human needs to decide this". Two entries is not much, but a queue that never empties stops being read.
All 13 integrity checks pass.
07:40 pm AWST · 431adc2
Entry 20 left this open as the obvious next fix: 8,770 live listings carry a model-year code, colour, body type or transmission inside the badge, and those cars can never match another car's trim. The plan was to strip them and see.
Measured on held-out folds, 46,172 predictions across the 30 largest trim-varying models:
shipped (normaliseBadge) $1,519 + model year, transmission, body $1,517 -0.2% + colours $1,517 -0.2%
Noise. Not shipped.
Two reasons it does so little. The 5.3% is concentrated in a handful of sources rather than spread through the corpus, so it rarely reaches a model with enough cars to measure. And badge levels already do the work string matching was being asked for: a comparable of another trim is moved onto the subject's by its measured price difference, which makes exact agreement matter much less than it appears to.
The risk was always an invisible mismatch — merging a Black Edition with a black car — and paying that for 0.2% would be a bad trade made on a hunch. Keeping the harness so it can be re-run if an extractor change ever makes badges dirtier.
07:40 pm AWST · e56c5e3
Half fixed, half measured and left open, which is the honest split.
live listings carried a trim nothing could match. Stripped and backfilled.
White 8 Speed Sports Automatic Wagon". Across 164,491 badged live listings, 8,770 (5.3%) carry a model-year code, colour, body type or transmission in the badge, concentrated in a few sources: zaimotors has 177 distinct badges for 178 listings.
Not fixed here because it has no safe handle. These tokens share a vocabulary with real trims — Sport is a descriptor in "M Sport" and a trim on a Ranger, Black Edition is a trim and Black is a colour — and normaliseBadge is conservative precisely for that reason. Loosening it trades a visible duplicate for an invisible mismatch.
Written down with the measurement and the method: extend eval-badge.ts, which already compares badge strategies on held-out folds, and ship only if it beats what ships. Same bar the trim restriction cleared, which is why that one can be trusted.
07:40 pm AWST · d5ca718
cars4us publishes its trim as "110TSI Comfortline For Sale ID72363". The id is unique per car, so **857 of its 881 live listings had a badge nothing else could ever match** — 857 distinct badges where 378 real ones exist.
A trim that matches nothing is worse than no trim. The valuation weights an exact-badge comparable at 1.5 and everything else at 0.8, and restricts to the subject's own trim once it has eight of them — so these cars were penalised against their own identical twins, and could never form a cohort.
Found while looking into why a 2023 Cupra Ateca VZX was hidden from the year picker: its 2022 sibling is recorded "Vzx For Sale ID67438", which is the same defect on a different source.
Stripped at ingest in tidyModelString and backfilled across vehicle. Deliberately narrow — the pattern is anchored at the end and requires the literal "for sale id" followed by digits, because badge text is where real trims live and a greedy cleaner here would eat one. Verified it leaves "GLX (4WD) 5 Seat", "Sale" and "For Sale" untouched.
cars4us now has 378 distinct badges across 880 listings, and its cars can find each other.
07:40 pm AWST · 7707002
A 2023 Cupra Ateca VZX listed at $33,000 was valued at $45,868 and labelled **high confidence, 0.76**. Reported by Taj.
Two faults behind it.
**The picker hid the year.** Years were narrowed by the selected badge, and the same trim is spelled differently across years because dealers append series and model-year codes: the 2023 VZX is recorded `Vzx KH`, the 2024 `Vzx`, the 2022 `Vzx For Sale ID67438` — a listing title leaked into the badge field. Choosing VZX therefore offered 2024 and 2022 and silently dropped 2023, so the only way to ask about the car was to describe it as a year newer. Narrowing badges by year is right and stops the picker offering a 2019 Wildtrak; narrowing years by badge hides data we hold. Years are now model-wide.
**Confidence was flattered by its own blind spot.** The three subscores ask how much evidence there is, how much it agrees and how fresh it is. When the kilometre slope cannot be fitted, all three are inflated by the same fact — a cohort of nearly-new cars is numerous, tightly priced and recently listed. Every 2024 Ateca sits at ~3,400km; the subject had 65,000; with no spread there is no slope. The workings said exactly that ("comparable set too small or km spread too narrow to fit a slope honestly") and the label said the answer was trustworthy. Widening the range was right and was not enough: a wide range with a confident label still reads as a confident answer.
An unadjusted estimate is now trusted in proportion to how little correcting it needed — exp(-gap/60,000km), multiplied through rather than averaged in, because it is a discount on all the evidence and not another kind of it. Exactly 1 whenever the slope was fitted, so the ordinary case is untouched.
Ateca 2024 VZX @65,000km high 0.76 -> low 0.30 Ateca 2023 @65,000km medium 0.54 -> insufficient 0.22 RAV4 2022 @60,000km high 0.80 -> high 0.80
Sanity suite unchanged, largest gap 5.5%. The user also now sees why: "the comparables average 55k km away from this car and no odometer slope could be fitted, so nothing corrects for that difference".
07:40 pm AWST · 4476486
The body drop finished — 181,059 cleared, 31 failures, all of them objects that were momentarily unreadable and correctly left with their body_gzip intact. 220,529 rows now hold their bytes only in object storage.
23,125 remain in the old column: rows the crawler wrote during the pass that was still running the old code. Collecting them means restarting the upload loop, and a restart begins with an empty cursor and scans past every row already migrated before it finds work. That is the query that timed out at 3,400 rows the first time, now with 220,529 rows in front of it.
So the upload queue gets the same treatment 0096 gave the drop queue: a partial index over exactly the rows still waiting. Both shrink as their queue drains and can be dropped once no row anywhere holds a body_gzip.
07:40 pm AWST · 0deb834
174,914 live listings across 698 dealer sources, 766,585 observations, 11,276 price changes, 5,363 delistings, 89 makes, 1,492 models. The morning's figures were 9.5 hours old and understated the corpus by 13,836 listings — a day's crawling with the raised caps.
07:40 pm AWST · 8d3960e
Records the time bound on maxDetailPages in the runbook: why it exists (7-8 seconds a page is the site's latency, not ours, so stock-only sizing wrote 24-hour budgets), that it is the only rule permitted to lower a cap, and the two thresholds that stop it churning configs.
07:40 pm AWST · 55b405a
An hour after setting seven caps, fit-detail-caps wanted to move them 1,750 -> 1,800: a config write and a line of report for about three minutes of extra fetching. The lowering path already required a source to be meaningfully over its limit before acting; the raising path had no such threshold and would nudge every cap up to the time bound fifty pages at a time.
A raise now has to clear 100 pages or 10%. Eight proposals become three, and the three left are the ones that matter: osborneparkkia 400 -> 600 (956 URLs found, 400 held after one crawl), westsideauto and bartons 100 each.
The point is that the report should list changes worth making. A tool that proposes noise trains you to apply it without reading, which is exactly when it will propose something that is not noise.
07:40 pm AWST · 2154924
fit-detail-caps sized every budget against stock and never asked how long that would take. Measured across finished runs, most dealer sites answer in **7 to 8 seconds a page** — their latency plus our politeness, neither ours to change — so the caps I wrote this morning were:
tonywhitegroup 9,800 pages 24.2 hours brisbanecars 8,150 19.9 countrycars 6,200 14.9 booran 4,350 9.7 johnhughes 3,950 9.3
Twelve sources were over four hours, and the current pass has been sitting on its tail for six of them: tonywhitegroup 3,826 pages done of 9,800, another ten hours to go, while 690 other sources wait for the pass to end so they can be re-observed.
So the cap is now bounded by time as well as stock: four hours at the source's own measured rate. It costs those sources nothing over time — they are rate-limited either way and the detail rotation takes successive slices across passes — it just stops one yard deciding when everyone else gets looked at again. Net -18,272 pages per pass, which is not lost coverage but the same fetches spread over shorter passes.
This bound may lower a cap, which nothing else here may do. The rest of the sizing is judgement and a hand-set number deserves to survive it; a budget that cannot complete in a day is not judgement, it is an arithmetic error, and this one was mine.
Hysteresis at 1.25x, because the first version trimmed klosters from 2,000 to 1,900 for being 4.2 hours instead of 4.0 — churning a config to buy twelve minutes, on a rate averaged from a handful of runs. Five hours is over; four and a bit is not.
Takes effect next pass: the running one read its caps at source start.
07:40 pm AWST · fcd4207
valueCar fetched comparables with `.limit(400)` and no ordering, so for any model with more than 400 live listings it got an arbitrary 400 — 86 models, 71% of the corpus. A 2022 Outlander was valued from 9 of the 241 that exist.
Recorded because of how it stayed hidden and what made it critical. A biased sample of a large set still lands near the middle when every comparable is corrected onto the subject and averaged, so the error was real and small and nothing pointed at it. 0.7.0 — restricting comparables to the subject's own model year, a change that cut held-out error 19.3% and is right — turned that mild bias into the entire basis of the estimate. I shipped it without asking what fed it.
The tell was sitting in sanity-valuations.ts for hours: worst case 7.9%, inside its own 10% threshold and easy to read as ordinary dispersion. Asking why rather than accepting it is what found this.
Two rules out of it: a limit without an order by is a sampling decision, not a safety valve; and a change that narrows what evidence is used makes every upstream sampling flaw newly load-bearing.
07:40 pm AWST · 88c262a
The check compared an estimate that excludes government charges against a raw median ask mixing drive-away ads with excl-charges ones. A drive-away ad in NSW carries roughly 3% stamp duty, so a cohort half advertised that way reads about 1.5% high before the model has done anything at all — part of every gap it reported was the yardstick, not the model.
The cohort rows now come back and are restated with the same priceExcludingDuty the valuation itself applies. Of the twelve cases, every one had drive-away ads in it: Ranger 152 of 737, Hilux 72 of 397, RAV4 54 of 356.
**The honest numbers are slightly worse than the ones I reported an hour ago**, and that is the point: Outlander +2.0% -> +4.6%, X-trail +0.5% -> +2.3%, Model Y 2023 -0.1% -> +1.9%. Nothing regressed; the comparison stopped flattering itself. Mean absolute gap 2.5%, largest 5.8%, and a mean *signed* gap of -0.1% — no systematic bias in either direction, which is the thing worth knowing and which the old measurement could not have told us.
07:40 pm AWST · b45bc41
The comparable query was one `.limit(400)` with no ordering, so for a popular model it returned an arbitrary 400 rows out of thousands. of 158,540 — 71% of the corpus**. A 2022 Outlander valuation was working from **9 of the 241 that exist**, because nine happened to fall in the slice.
That was survivable while every comparable was corrected onto the subject and averaged; a biased sample of a large set still lands near the middle. It stopped being survivable in 0.7.0, which restricts to the subject's own model year when it has at least eight. The restriction is only ever as good as the sample of that year, and this one built an estimate from nine arbitrary cars while 232 sat unread. I shipped that restriction this morning without checking what fed it.
Now the subject's year is fetched on its own and in full, with neighbouring years fetched separately and bounded — their job is to measure the shape of depreciation, not to outvote the subject's year. The largest make/model/year cohort in the corpus is 1,890 and only twelve exceed 1,000, so the 1,000-row limit reaches all of all but twelve, and those are new-car cohorts the used gate discards anyway. The two run in parallel, so latency is the slower of them rather than the
Twelve cohort-median cars against their cohort's median ask:
Outlander 2022 +7.9% -> +2.0% Corolla 2021 -4.7% -> -1.9% X-trail 2021 +1.6% -> +0.5% Model Y 2023 -0.6% -> -0.1% Hilux 2021 (395 comps, now resolves)
Mean absolute gap 2.57% -> 1.98%, worst case 7.9% -> 5.6%.
Found by asking why the worst case in sanity-valuations.ts was 7.9% rather than accepting it as within tolerance. The answer was not the model; it was that the model could not see.
07:40 pm AWST · 6644c96
Two things, both about a number being believed for longer than it should be.
**The index drop the ledger never heard about.** `live_snapshot` created three indexes and ad-hoc SQL dropped them, so the migration record and the catalogue `pg_stat_statements` still shows each create and its matching `drop index if exists` at one call each. 0110 records the drop, idempotently, and puts the reasoning on the table comment so nobody restores them from the plan alone. They went because every consumer aggregates the whole 44MB table, so a sequential scan is the intended plan and an index only cost write time on the truncate-and-insert.
**The cadence the pages could not see.** The board list was a `VALUES` block inside `check_stale_boards()`. The integrity suite therefore knew each board's refresh cadence and the pages drawing from those boards did not, so a board whose refresh had stopped kept rendering in the present tense on the site while the staleness was reported to a checker nobody reading the page runs. That is the same shape as every other defect this project has found: the check existed and fired into an empty room.
`board_spec` holds it once and both sides read it. `check_stale_boards()` keeps its exact signature and columns, so `check-integrity.ts` and `health.ts` are untouched. `board_status()` answers the same question on the request path, optionally for one board so a page pays for what it draws from rather than counting all nineteen.
Four states: `fresh` draws no dateline, because a dateline on every number all the time is noise a reader learns to skip. `ageing` and `stale` date the figure and `stale` says the refresh has not run. `empty` separates a board that should hold rows from one where holding none is the honest answer, which is the distinction that matters: an empty `markdown_board` means no cohort had 30 listings showing a cut, an empty `market_make_board` means something broke.
Nothing is ever hidden because its board is old. A six-hour-old median is still the best answer we have and withholding it would be a worse lie than dating it.
Proven by forcing each transition and restoring the exact prior values:
- `fresh` -> `ageing` -> `stale` on `aged_board`, then back to `fresh` - `empty` on `ladder_event`, which is genuinely empty, with `check_stale_boards()` flagging it at `must_have_rows` true and not flagging it at false. Probe row removed, `board_spec` back to 19 rows - end to end in the browser on /market: no dateline while fresh, the dated qualifier once `market_state_board` was forced stale, and gone again after restoring. 13/13 integrity checks before and after
The `asAt` stamp is built from two `toLocaleString` calls rather than one, because a single call puts a comma between the date and the time and "14 Aug, 00:25 UTC" reads like a list rather than a timestamp.
11:42 am AWST · 22ac6f3
The duplicate-URL check found 277 URLs live under two source ids again on 2026-08-13, twelve hours after the merge that was supposed to end that, and 403 rows were retired to clear it. This is where they came from.
200 rows appeared under dealer:cherymoorooka and 200 under dealer:cheryspringwood at 16:46 and 16:49 — hours after 0083 merged both into dealer:motoramachery, disabled them, and asserted neither held a live listing. Nothing has been written to either since, so it was a single run rather than a leak: all three ids point at motoramachery's own sitemap, so whatever walked it created the same page three times.
Three defences all missed it for the same reason. crawl.ts only walks sources where enabled is true. 0082's constraint refuses to enable a source that still names its survivor. Both govern *reaching* a source through the scheduled path, and neither stands between a source id and observeListing — the only function that creates a listing row. Anything holding an id could write to it.
So observeListing now refuses, naming the survivor in the error. Creation row that records the same page, which is the correct reading rather than a resurrection. The source lookup is cached per process and folded into the existing state fallback, so this costs one query per source per run and removes one per new listing.
Verified end to end rather than reasoned: observeListing called against dealer:cherymoorooka with a URL not in the corpus threw "was merged into dealer:motoramachery; refusing to create a listing under a retired source id", and the row count for that URL was 0 before and 0 after. A live source returns mergedInto=null, so the guard does not fire on the normal path.
The distinction this records in DATA-QUALITY 14: 0084's check is a monitor, this is the defence. A check that reports the same breach every morning is a subscription to the problem. Third time in that file a rule existed, was correct, and was never in the path that mattered.
Left open deliberately: the guard is application code, so a direct insert still bypasses it. The trigger version is the structural fix and belongs in a migration — not written now, because three sessions were live against this database and a numbering reconciliation was already in progress.
11:29 am AWST · fb55966
**277 URLs live under two source ids.** 274 of them were one cluster: cherymoorooka, cheryspringwood and motoramachery all configured with motoramachery's own sitemap as their indexUrl, so three sources created the same listing three times. My first reading was that merged sources were still being crawled; all 24 are in fact already disabled, which makes these leftovers from before that — and worse than ordinary duplicates, because a disabled source is never re-crawled and so its listings can never be delisted. They would have stayed live forever.
403 rows retired to `merged`, each naming its survivor. The first attempt did not, and the schema refused it — listing_merged_names_its_ survivor is a CHECK that status = 'merged' implies a survivor, and it caught exactly the sloppiness it exists for. Live 170,178 -> 169,787, duplicate URLs -> 0.
**The second finding was the check's own fault.** It reported a Fiat Ducato and an LDV Deliver 9 "held out of scope by nothing but a past UPDATE". I nearly reversed both on the documented policy that vans are kept deliberately — then read the rows: badge `Motorhome`, badge `Campervan`, body_type `truck`. Every one excluded by a rule that exists.
The check applied isNonCarMake and isNonCarBodyType and not isNonCarBadge; it did not even select the badge column, while ingest excludes on all three (`outOfScope` in store.ts). So it was manufacturing findings about correctly classified data — worse than missing them, because a finding gets acted on. It now applies the same three rules as the thing it checks.
All 13 checks pass.
10:55 am AWST · f9e5d06
Upload finished: 182,839 bodies in object storage, 6 failures, all of them transient gateway errors whose rows kept body_gzip and never got a key — intact, just unmigrated. 500 of 500 verified byte-identical by sha256 before anything was cleared.
Then the drop pass, which was wrong in two independent ways.
**Writing.** One UPDATE per row, each rewriting a row that carries a 45KB TOAST value while the crawler competes for the same table: 0.4 rows/s, which is 130 hours for 187,839. Now the batch is verified concurrently and cleared in a single statement — fifty downloads and one write. A failed batch clears nothing and is picked up next run, so nothing is ever half-done.
**Reading.** The scan could not find its own work. The predicate looks selective and is not: 187,839 of 574,000 rows match, but the oldest captures come first by id and are mostly ones whose bodies were deduplicated away long ago, so it removed 23,942 rows by filter to find 50 — 24,168 buffers, 1,562ms, a timeout on the first query of the run. A partial index over exactly the rows still to clear takes it to 0.300ms and 50 buffers. It shrinks as the work proceeds and vanishes when the migration is done, which is the pleasant property of indexing a work queue rather than a table.
The read-back before clearing stays. It is the difference between believing an object is there and having just read it, and what it guards is the only remaining copy of pages that can never be re-crawled. Four hours in the background is a fair price for that.
09:59 am AWST · 87301f3
Where bodies live, why every reader must go through rawBody(), why writes are object-first-row-second and must not be reordered, how to run and resume the migration, and what each failure mode means.
The three that matter: verify before --drop, because it removes the only other copy of pages that can never be re-crawled; VACUUM FULL takes an exclusive lock and belongs between passes, not during one; and an orphaned object is harmless while a row pointing at a missing object is not, which is the whole reason for the write order.
Re-parse verified end-to-end through the new path: rawBody reads storage byte-identically (200/200 by sha256) and reparse.ts reads through rawBody — 5 listings re-parsed from 5 captures, none without a retrievable body. The capability that justified the migration still works.
09:32 am AWST · ba628a9
While a row still carries both body_gzip and a body_key, the migration can be checked rather than trusted: read it back through rawBody(), which prefers the object, and hash against the body_sha256 recorded when the page was first fetched. A match proves the object holds exactly the bytes the crawler saw, judged by a hash written before object storage existed.
200 of 200 byte-identical.
This is the gate for --drop, and it needs to be. Clearing body_gzip removes the only other copy of a page that in many cases can never be re-crawled — the listing is delisted and gone — so "the upload returned no error" is not a good enough reason. Refuses with a non-zero exit if any body fails to match.
Sampled rather than exhaustive on purpose: it reads whole bodies out of TOAST *and* out of storage, so checking all 182,071 would cost more than the migration itself, and the drop pass re-reads every object it clears anyway. This answers the different question — whether the two copies agree — which a sample answers well.
09:02 am AWST · 48fdc4a
probe-candidates returned "WAF challenge on inventory pages" and stopped there, ahead of the sitemap fallback. The reasoning was that a site refusing our guessed paths refuses everything.
canningvaletoyota disproved the reasoning: a challenge on the guessed paths, and a sitemap advertised in its own robots.txt naming 66 vehicle URLs that fetch without complaint. Its robots.txt is also explicit that we are welcome — Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php, which is a site owner deliberately permitting the endpoint its own pages call.
The fallback's existing comment already made this argument about 404s — "a guessed path answering 404 says nothing about whether a dealer publishes its inventory" — and a challenge on a path we invented says just as little. The WAF verdict survives as the last word rather than the first, and now reads "and no usable sitemap", which is the thing that earns it.
**It did not rescue this source.** canningvaletoyota's detail pages are client-rendered too: 200, no price, no JSON-LD, nothing to parse. The probe was right to reject it and wrong about why, and only the second half is fixed here. Its verdict now reads "jsonld recognised at /vehicles but it lists no vehicles", which points at the real problem.
Worth having anyway: the ordering was wrong independently of whether this particular site was behind it, and two sources have been rescued by that fallback before.
08:41 am AWST · a139b31
Second sweep of the marketplace-name channel returned 4 of 91, against 17% on the first. Same channel, and the fall is not noise: the first sweep took the dealers whose trading name is their domain, and the tail is names that are not — "CarPlace", "First Choice Freo", "Next Ride Malaga". Guessing a domain from a business name stops working before the names run out.
The more useful finding came from fixing the probe's verdicts. Once "never found the page" and "reached it and could not read it" stopped being the same message, the rejects turned out to share a shape: canningvaletoyota (a real dealer with 1,000+ cars), carplace, coastlineautos, morleycityhyundai and eliteautodeals all answer 200 on their stock page with no vehicle data in the HTML — no JSON-LD, no listing links. carplace is Astro, morleycityhyundai is WordPress, so this is not one platform missing one extractor. It is a rendering style, and it is becoming common.
Seeded tradeprestige (68 URLs) from this sweep. Eight sources today.
07:59 am AWST · 8709a6b
The sameSite fix landed: autopact reports found=3250 where every previous pass found 24, 200 of 200 attempted parsing at 100%. Its cap was never raised to match, because fit-detail-caps ran before the fix when the source still looked like it had 24 URLs. Raised 200 -> 2000 by hand with the evidence recorded on the source row.
Then tried to generalise it so the next unlocked source needs no human: trust `found` when a source parses >=95% of what it attempts. Added the parse rate to source_max_found, wired it in, ran the dry run — and it proposed lifting pattersoncheney from 700 to 4,850.
That is the exact waste this script exists to prevent. pattersoncheney finds 3,872 URLs and holds 766 cars after many passes of rotation; the other ~3,100 are demonstrably not cars. Its parse rate is 99% anyway, because parse rate is measured over pages *attempted* and a source only attempts URLs its extractor recognises. It says nothing about the URLs found and never fetched, which are the ones the question is about.
So the heuristic is reverted rather than shipped, with why it fails written where the next person will reach for it. What actually separates autopact from pattersoncheney is coverage history — how much of its found set a source has had the chance to fetch — and that deserves measuring properly rather than approximating with a number that happened to be to hand. The ratchet handles it meanwhile: live grows, the cap follows, a few passes instead of one.
parse_rate stays on the RPC. It cost nothing, it is real, and it will be wanted when coverage history is done properly.
Caps applied with the correct rule: pattersoncheney 700 -> 1,100 (766 live plus headroom, not 4,850), tonywhitegroup 7,500 -> 9,800, cranbournetoyota 450 -> 700, blackburnkia 200 -> 300. +1,910 pages/pass.
07:43 am AWST · 3ccfb77
Six refresh jobs each scanned the corpus, five through `listing_current` and its two laterals. `refresh_depth_boards` paid twice on its own: `_cur` through the view, then `_md` re-running the same observation lateral purely to reach `displayed_previous_price`, which the view does not project.
`live_snapshot` does that once. Live listings only, mirroring `listing_current` exactly including the merged exclusion and the `fields_empty` back-lookup, plus the parsed values the boards read and an `advertised_days` resolved against one clock so every job in a cycle agrees what "today" meant.
**The scan was never the expensive part.** The first version blew the two minute ceiling, and EXPLAIN put the blame somewhere I had not looked: the nested loop finished in 436ms on a 1-in-33 sample while the aggregate above it took 3,843ms, with 122,000 buffer hits the scan had not touched. Those are detoast reads. Every mention of `coalesce(f.fields, o.fields)` fetches and decompresses the whole payload again, and the query named it fifteen times per row: three each for `views`, `dapprice` and `egcprice`, and six more inside `advertised_days`, which reads two keys three times each and is a SQL function, so it inlines into the caller.
`jsonb_to_record` cut that to one detoast and the pass to 109s. Still wrong, and the full plan said why: 93 of those seconds were that single remaining detoast, 115,701 block reads. Only 40,918 of 161,134 current observations carry a payload at all and those total 172 MB, but they sit scattered through a 1,642 MB toast table shared with every historical observation, so reaching 172 MB of data costs about 900 MB of reads. It was growing with the history, not with the number of live cars.
So the five values are extracted when the row is written and `fields` is still in memory and untoasted, by the trigger that already sets `fields_empty`. The refresh reads narrow columns and never touches the toast. **21 seconds.** Depth boards 7m58s to 6s. The daily rollup had failed on its full 15 minute ceiling on both 12 and 13 August, so the daily series was being written by nothing but ad hoc runs; it now takes 7 seconds.
Verified rather than reasoned about, because this rewrites what every board is built from:
- all 223,952 non-empty observations: every `f_*` column byte-identical to the `fields->>key` it replaces, so the expressions could stay untouched and no board can move - 156,116 listings compared column by column, `live_snapshot` against `listing_current`: zero differences on every time-independent column, and `advertised_days` off by exactly the 0.1 that 15 minutes of elapsed clock rounds to on its `createdAt` branch - boards rebuilt against the previous function's output: identical but for eight cohorts whose counts had moved by one to three listings in the three minutes between runs, every one downward, which is the corpus growing - the two halves of the rollup that could not move to a live-only snapshot, checked against the view directly: exits agree at 2,445 listings and $71,194,413, events across all 401 make and model groups
Guards, because the failure mode here is silence rather than an error. Both consumers refuse a snapshot older than two hours, and the truncates are in the same transaction, so a refusal leaves the previous boards standing instead of replacing them with thin ones. `refresh_live_snapshot()` refuses while any observation is still unextracted, since an unextracted row reads as null rather than as a fault. `live_snapshot` joins `check_stale_boards()`.
`observation_pick_current_idx` stores the ordering expression rather than `source_view`, though only 'detail' and 'index' exist today and 'detail' happens to sort first. A plain column index would work and would quietly stop meaning "detail wins" the moment a third view name appeared. It and `observation_facts_todo_idx` were built CONCURRENTLY against the live database and so recorded in 0105 rather than left as catalogue objects with no migration behind them, which is how 0088 went missing.
07:30 am AWST · 897dbdc
Two failures, same root, found by running it rather than reasoning about it.
The loop asked for "the next rows with a body and no key, ordered by id" each time, which re-scans every row already migrated to find the next batch. Each batch costs more than the last. It ran at 19/s to 3,400 rows and then died on `canceling statement due to statement timeout` — fast, then suddenly not, which is how this always presents.
A cursor fixes the loop and moves the problem to startup: a restart with an empty cursor has to scan past everything already done before it finds work, and that first query times out for the same reason. `raw_capture.id` is a uuid, so there is no "highest migrated id" to infer the position from — the order is effectively random and the only record of where we got to is one we keep. So the cursor is written to disk after every batch. The --drop loop had the identical flaw for the identical reason; fixed there too rather than waiting to rediscover it.
Batch size down 200 -> 50. The cost is the payload, not the row count: each row drags ~45KB of TOAST, so 200 is a 9MB response that times out against a busy database while 50 sustains the same 18 rows/second.
07:23 am AWST · c852794
raw_capture was 8,654 MB of an 11 GB database — 79%, almost all TOAST. The disk bill was the least of it: every body went through the WAL and into every backup, 8.5 GB of blobs that are read approximately never competed for a 512 MB shared_buffers with the rows the site queries, and the largest-payload insert was the first to fail under load. That is the write failures the 2026-08-14 pass logged, climbing 0.15% -> 0.32% as the table grew.
**The reason to do it is not the size, it is what the size was costing.** shouldRetainBody kept one body per URL, so a page that changed after its first capture had the new version discarded — including the version where the price changed, which is the one a re-parse would most want. 290,114 distinct bodies exist; 182,071 were kept. 37% of the raw record thrown away to hold down a table that should not have held it, against a SPEC promise that a better parser can re-read old captures without re-crawling.
The old rule even contained its own refutation: "the second capture of an unchanged page is already deduplicated by sha". If unchanged pages cost nothing, the rule was only ever dropping changed ones.
Bodies are already content-addressed by sha256, which is exactly the object-storage access pattern — the key *is* the hash, identical bytes collapse to one object with no bookkeeping, and an upload is idempotent, so unlike the database write path this one is safe to retry. Retaining every distinct body costs ~13 GB against the 100 GB Pro includes.
Shape of it:
body-store.ts put/get/exists, keyed sha256, two-level fan-out store.ts object first, then row. A row pointing at a missing object reads as data loss; an object with no row is a few kilobytes nobody references. Order the failure onto the harmless side. raw.ts four fallbacks: own object, own legacy column, sibling object, sibling legacy column. Spans the migration and keeps working afterwards for anything unmoved. shouldRetainBody now returns true and the query it needed is gone — one fewer round trip per detail page, on the path that was timing out.
Verified round-trip before touching real data: object written, body_gzip null, bytes identical on read-back, and a second capture of identical content resolving to the same key.
The backfill is separate and resumable, and --drop is a separate flag is cleared. It ran at 2.5 rows/s serially — twenty hours — because each row cost three sequential round trips; dropping the sibling lookup (the upload answers that question by doing it) and running eight at a time puts it at 17.8/s.
Migration is additive, so the old code kept running against it untouched while this was written.
07:08 am AWST · b32f615
raw_capture is 8,654 MB of an 11 GB database — 79%, and almost all of it TOAST rather than index. 571,141 capture rows, 181,907 with a body at roughly 48KB each. That is the design working: one body per listing rather than one per observation, which is what SPEC's re-parse promise costs.
It surfaces as write failures. Through the 2026-08-14 pass they climbed 0.15% -> 0.23% -> 0.32% of extractions, almost entirely `raw_capture insert failed`, with occasional `observation insert failed: <!DOCTYPE html>` — a gateway error page, which is what a timeout looks like from the client. Not host-specific, not size-specific, clustered in often as the table grows.
Nothing corrupts. Failures are counted rather than dropped, cannot produce a false delisting, and the listing is re-observed next pass.
Writes down the decision rather than making it: storage grows ~48KB per listing on a project committed to donations-only funding, and every lever trades against SPEC's retention promise except moving bodies to object storage, which keeps the promise and is the largest change. Taj's call.
07:04 am AWST · c6b0d46
Grouping live listings on their punctuation-stripped model name found 39 models carrying more than one spelling. Most are single-digit and harmless. One is not: **163 of 481 GLC-class listings sat under "GLC Class" while the other 318 sat under "Glc-class"**, so a GLC valuation drew on two-thirds of the comparables that exist and had no way to know it. Mercedes-Benz accounts for most of the rest, because every one of its ranges is a "<letters> Class": GLE 45, GLA 31, C 30, A 26, GLB 16, CLA 14. Outside it, Omoda 5 (27) and F-150 (22).
Written out rather than derived by collapsing separators, which is the obvious general rule and is wrong twice in this very list: LDV's "G10" and "G10+" are different vans, and Lexus's "RX450h" and "Rx450H+" are a hybrid and a plug-in hybrid. That rule would trade a visible duplicate for an invisible mismatch, which is the worse of the two.
Targets are the spelling each model already predominantly uses, so recanonicalise moves the minority across rather than churning every row. It runs in the cron chain after each crawl, so this applies to the existing corpus as well as new ingests.
Also adds sanity-valuations.ts: twelve common cars, each placed at its own cohort's median odometer, estimate against that cohort's median ask. It is not a replacement for the held-out evaluation — a model returning the same-year median would score perfectly here and ignore odometer entirely — but it answers the question people actually ask. Largest gap is Outlander 2022 at +7.9%; Model Y 2025 is +0.7% where it used to be 11% low.
The splits surfaced because that script reported "thin cohort" for Hilux and X-Trail, two of Australia's best-selling vehicles. The cause was my own casing, but chasing it found the real thing. A check that quietly skips what it cannot find will eventually skip the thing that broke, so the spellings are now the corpus's own.
06:59 am AWST · a9a0123
probe-candidates returned `inventory HTTP 404` for canningvaletoyota, a real dealer with 1,000+ cars whose /vehicle-inventory answers 200 on both apex and www. The 404 came from a different guessed path. I spent several fetches hunting for the right URL when the URL was never the problem.
That is the conflation the WAF branch immediately above already exists to prevent — "a site we may never crawl" against "a path guess that was wrong" — with a third case nobody had separated: reached the inventory page, could not read it. Only that one says anything about the site.
Verdicts now distinguish:
<path> answers 200 but no extractor recognises it <platform> recognised at <path> but it lists no vehicles inventory HTTP <status> (no path ever answered) no recognised inventory page (nothing to say)
Re-probed with it, three hosts that all read "no recognised inventory page" or "404" now read: canningvaletoyota is a WAF challenge, carplace and coastlineautos serve a platform we have no extractor for. Three different problems with three different answers, previously indexed under one non-fact.
/vehicle-inventory added to the path list too, on its own merits.
06:50 am AWST · cc7cfe4
Autotrader and CarsGuide print the selling dealer on every listing, which makes them a directory of dealer sites we have not found. As at 2026-08-14 that is 1,066 distinct dealer names across 13,549 marketplace listings against 700 sources, and 102 unmatched names carry 20+ live cars each.
This is the highest-volume of the three channels measured yesterday — 33 of 190, 17% — and the only one that prices the prize before you probe: the marketplace listing count is a floor on what that dealer's own site holds. It also does not run dry the way the sibling guess does, because marketplaces keep listing dealers nobody has looked at.
The name is not the domain, so several spellings are emitted per dealer and the probe sorts them out. "Purely commercials Bibra Lake" is purelycommercials.com.au; "Osborne Park Kia - Used 2" is osborneparkkia.com.au — groups name a site after the franchise, not after the lot.
Matching leans toward excluding candidates: a name counts as known if it prefix-matches a source name or hostname either way. That drops some we genuinely lack — "Melville MG" is excluded by melvillemazda — but a false exclusion costs one missed candidate while a false inclusion costs a probe against a site we already crawl, and probes are the scarce thing.
First run: 2 of 23 usable. osborneparkkia (950 vehicle URLs) and purelycommercials (19), both seeded. Most of the rest resolved to nothing, a 404, or a WAF, which is the argument for probing before seeding rather than after.
Six sources seeded today: four from the sibling guess, two from here.
06:47 am AWST · f06ec27
Charts now declare the evidence they need, a scheduled evaluator measures every cohort against the declaration, and the resolved rung decides what each may draw. This is the mechanism DEPTH-PLAN's unlock table wanted and never had: the pre-registered Tell was written to publish at 2,000 observed cuts, that passed days ago at 5,997, and nothing fired, because a gate written in prose is not a gate.
Schema in 0098, seeds in 0099, evaluator in 0100, invariants in 0101.
Three charts to start, chosen so the mechanism is proven rather than merely declared. price_vs_odometer and book mostly pass and discriminate; price_over_time must refuse for every cohort in the corpus. Thresholds are not invented: each repeats a bar this codebase already enforces somewhere, which is the point, since the ladder is where those scattered constants become visible and checkable.
Design decisions that earn their complexity. One comparator, measure >= threshold, so a requirement cannot hide an opinion in a where clause. First failure rather than highest pass, so a thin cohort cannot skip a rung it fails. Cohort keyed on make|model and not the slug, because 26 slugs map to two distinct pairs. Measures count vehicles and not listings, because syndication inflates a Ranger 2.2x and a measure counting adverts would promote cohorts that do not exist. A null block_have means unmeasured, which is not zero. Promotion waits for the evidence to hold twice; demotion is immediate.
Verified against the live corpus, 1,442 cohorts and 4,326 resolved states. Geely EX5 resolves to rung 1 on the scatter, blocked by km_spread_km at 5,686 against the 20,000 a robust fit needs, so it will not draw a line through a cohort with no spread. price_over_time is rung 0 for every cohort, blocked at 1 contiguous day against 6, which is the honest state of a corpus whose two rollup days do not touch.
check_ladder_spec asserts monotonicity, produced measures, ungapped rungs, resolved charts and evidence freshness, and runs in check-integrity.ts, which the crawl workflow actually executes. Proved by making it fail: relaxing a rung-2 threshold below its rung-1 counterpart produced "model.price_vs_odometer rung 1 requires priced_km_cars >= 3 but rung 2 requires only >= 2", and the threshold was restored.
Nothing reads the ladder yet. That is Phase 2.
06:44 am AWST · b30f368
The 2026-08-14 pass logged `raw_capture insert failed: undefined` repeatedly, on two unrelated hosts. That is the least useful thing a diagnostic can say: the write failed and the log cannot distinguish a payload too large from a gateway timeout from a constraint violation. One sibling failure arrived as `observation insert failed: <!DOCTYPE html>` — the gateway answering with an error page instead of JSON, which leaves `message` empty while `status` is not.
describeDbError reports code, status, message, details and hint, and falls back to the serialised object when a driver hands back something with none of them. Truncated, because one of these failures was a whole HTML document and a stack trace carrying a web page helps nobody.
Deliberately no retry. These are ~5 failures in 11,000 observations (0.04%), the listing is counted failed rather than silently dropped, it cannot cause a false delisting — the URL was in the index, and delistings come from index absence — and it is re-observed next pass. Observations are append-only, so a retry after a *lost response* rather than a failed write would insert the same observation twice and corrupt exactly the counts the append-only log exists to protect. Worth doing properly with an idempotency key, not reflexively.
06:31 am AWST · b2ac54c
Australian dealer groups name sites <location><marque>.com.au regularly enough that the gaps are guessable — knowing melvillekia and melvillemazda exist is evidence for melvillehyundai. suggest-siblings.ts emits the combinations we do not hold, for probe-candidates.ts to test.
Which locations are worth guessing is the measured part, and the reason the script defaults to recently-seeded ones. Over 224 probes on 2026-08-13:
siblings of a group found that same day 4/16 25% dealer names taken from a marketplace 33/190 17% siblings of a long-established group 0/18 0%
The zero is the useful number. Castle Hill, Frankston and Mandurah carry many marques each and had already been mined, so what is still missing there are the marques those groups do not sell. A group's gaps deplete once worked.
This run: 4 of 45 usable, 8.9% — below yesterday's 25% and above the 0% of established groups, which fits. Melville had five marques seeded yesterday and gave up two more; the rest of the batch was mostly a location parser reading group names like "western" and "valley" as places, which is noise the probe filters cheaply.
(21), melvillerenault (12). About 182 listings.
Nothing is written by this script. It prints candidates; the probe decides what is real, because seeding a WAF or an unsupported platform costs a crawl cycle and leaves a permanently broken row on /coverage.
06:17 am AWST · 1e3fd6b
revalidate alone did nothing. Both model routes still answered with cache-control private, no-store and x-vercel-cache MISS, because a dynamic segment without generateStaticParams is rendered fresh per request whatever its revalidate says. Declaring the function is what flips the route from dynamic to statically generated and revalidated; the build marker goes from f to a filled circle.
Both return an empty list on purpose. Seeding the twelve largest models was tried and reverted: Ranger, X-Trail and Tucson all time out at build, so the deploy baked their degraded page in and would have served it for the whole fifteen minute window. Prebuilding nothing means a cohort is only ever cached in the state it was genuinely in when somebody asked for it, and revalidation heals it once model_chart_board lands.
06:16 am AWST · 8948fe1
Entry 17 filed double-crawling a host as a risk the stale window made possible. It had already happened: market:autotrader has two overlapping run rows on 2026-08-13, 12:40 and 21:10, and late in the overlap the log records a 503 from www.autotrader.com.au. Per-host politeness is enforced per process, so two crawlers is two request streams and neither knows the real rate.
Exactly one source overlapped, not dozens, so this was a targeted run started against autotrader while the first was still fetching it — let through because the 12:40 row was eight hours old and the guard only looked back two.
Already fixed by the liveness change: an open row is a claim regardless of age, and reaping needs old-and-quiet. Recording it because the difference between "this could happen" and "this did happen, here is the 503" is the difference between a note and a lesson.
06:12 am AWST · 6b6e815
autotrader answers in about four seconds a page. There is no Crawl-delay in its robots.txt and our own minimum is one second, so the site's own latency is the limit and we are already slower than we ask to be. At a budget of 8,000 that is a nine-hour run for one source, and it runs alone — the other 689 finished hours before it started.
The ordering pins it there. A pass sorts stalest-first on started_at, so whichever source starts last carries the newest timestamp and sorts last again next pass. Harmless for a small source; compounding for a large one, because nothing is in flight to overlap with and its hours are added to the pass rather than absorbed by it.
Lowered to 3,000, about three hours. It costs autotrader nothing over time — it is rate-limited either way, and the detail rotation takes successive bites of its ~11,000 listings across passes. What it buys is the other 690 sources being re-observed sooner, which is where price changes and delistings come from.
The proper fix is scheduling: dispatch the longest sources first so their hours overlap rather than trail. Not done here, because the current order also protects the least-covered sources when a pass is interrupted, and trading that away deserves measurement rather than a 6am guess. Written down in OPERATIONS with the inputs it needs.
06:11 am AWST · e41d22d
A pass is ordered never-crawled first, then stalest, so an interrupted run costs the least-covered sources nothing. That order was built by selecting every crawl_run and reducing client-side, with no limit and `const { data } = await`.
crawl_run is at 4,990 rows across 722 sources and grows by ~690 a pass. PostgREST caps a response at 1,000, so the reduction ran over the newest 1,000 runs and 12 sources were absent from it entirely — read as never crawled and sorted to the front. The count printed at the top of every pass was wrong by the same 12: it would have said ~20 never crawled where the real number is 8.
The truncation is the mild half. The unread error is the other one: a timeout on an unbounded scan arrives as null, every source then looks never-crawled, and the order collapses to alphabetical — precisely the failure this rotation exists to prevent, which once left 88 of 183 sources never visited. Silent, and it would look like a crawl working normally.
So the aggregate moves into SQL, bounded by source count rather than run count, and a pass that cannot read its own history now throws instead of crawling in an arbitrary order and starving its tail. DATA-QUALITY entry 16 is the class; this is the fifth instance and the first in the scheduler.
06:00 am AWST · c526e56
161,078 live listings across 690 dealer sources, 589,888 observations, 7,950 price changes, 4,439 delistings, 88 makes, 1,455 models. The previous figures were a day old and understated the corpus by about 9,500 listings.
05:55 am AWST · 5b855f7
A 2025 Model Y read $58,582 against a $65,888 median ask. The comp set held 16 cars from 2025 and 269 from 2022-23, and the year steps are not plus 2025 stock averaging 21,000km against 52,000.
The year term already knew this. The cap stopped it acting: moving a $40,480 2022 car to 2025 needs +$25,400, and the shift is capped at 35% of ask, $14,168. It landed at $54,648, and 114 such cars outvoted the 16 that needed no adjustment at all. This is the trim problem from 0.6.0 in a different clothes, and it takes the same answer — a year with enough of its own listings is compared against itself.
Held out over 43,337 cars in 30 year-varying models: **median error -19.3%**, and **-32.0%** on subjects whose model year jumped more than 20%, which is the shape this is for.
Model Y 2025 @21,000km $58,582 -> $65,888 (median ask $65,888) Pajero Sport GLX 2022 $42,000 -> $37,193 (listings $35-38k)
Two things measured and rejected, worth recording because both looked better than they were:
- Down-weighting comps whose shift was capped, in proportion to what was clipped off, is the more elegant idea and does nothing: -0.7% overall, and 2.8% *worse* on the very case it targets. - Restricting unconditionally scores -22.1%, better than what shipped here, but part of that is survivorship. It leaves 287 of 43,337 subjects with no estimate at all, and those are hard cases its average no longer has to carry. Guarding the restriction so it gives way rather than empty the set answers every subject the old model answered and still gets -19.3%.
Restrictions nest by giving way in order — trim first, then year — since eight of a subject's year and eight of its trim can have no car in common.
The narrowing is now stated in the workings. An estimate built from 13 listings when the model shows 274 looks like a mistake until the page says it was a choice, and the user asked for the basis of these numbers to be legible.
One correction to my own measurement along the way: the first harness left restrictToTrim out of the baseline, so it was scoring year restriction against a weaker model than the one that ships and would have credited it with the trim fix's work too. Against the real baseline the effect is larger, not smaller — -22.6% where the flawed run said -12.0%.
05:34 am AWST · ba46785
05:34 am AWST · 39018c4
Three of the five Phase 0 items from CHARTS-PLAN.
/moving quoted an aggregate median and a "cut before selling" percentage whenever a filter matched a single sale. NT printed 43 days and 100% from five sales, where the median was noise and the percentage was five coin flips landing the same way. Both now answer to the same bar of 20 the home and market pages already use before quoting days to sell, and below it the page states the real count and says the sales are real while their average would not be. The per-model rows keep their existing bar of 4, where the count sits in the next column.
daily_model_rollup was keyed for its writer and not its readers: the only index led on day, so every per-cohort series would have scanned every day ever written. A covering index on (make, model, day) including the plotted columns makes it an index-only scan, verified with zero heap fetches. At 1,190 rows this changes nothing measurable, which is exactly why it is cheap to add before the table is large.
model_ticker was recomputed on every visit. Measured in production: Mazda CX-5 2.9s, Ford Ranger over the API role's 8 second ceiling and returning a 500. The page is now revalidated every fifteen minutes rather than force-dynamic, which takes every model that renders at all off the request path.
Caching cannot rescue Ranger, because a page that has never rendered once has nothing to serve stale, and this is what model_chart_board in Phase 2 exists to fix. What it must not do meanwhile is 404. A thrown timeout was falling through to notFound(), so the largest cohort in the corpus read as a model that does not exist. Unknown model and known model we cannot yet summarise are now different states, and the second says so and offers the views that do cover it.
01:09 am AWST · fed2731
Making the reaper require quiet fixed live runs being closed mid-fetch and introduced a way to hang. A pass that dies leaves open rows; anotherCrawlIsRunning() reads them as a live claim and refuses to start. Normally the corpus falls quiet within seconds of the death and the next scheduled pass reaps them — but a pass chained to start immediately after another begins while observations are still seconds old, sees "not quiet", reaps nothing, and refuses on rows whose process is long gone.
That is not hypothetical here: a pass is chained behind the current one so the raised caps and the sitemap fix land now rather than at 06:43.
So a row is reaped by either route. Quiet: older than two hours and nothing being written anywhere, which is the normal path and the one that spares a four-hour brisbanecars run. Hard: older than twelve hours regardless, a threshold no legitimate run reaches — the largest budget is 8,150 pages, about four and a half hours. Twelve hours is not a slow source, it is a corpse.
12:46 am AWST · 1c96dc8
The delisting alert was a false alarm with a real cause, and the class is new to this register: not a read that failed quietly (9-12), but a guard that documented its justification in terms of a number owned by another file. The comment was true when written and falsified by a config change that had no reason to know it existed.
12:45 am AWST · 21f8e9d
The 00:36 monitor alerted on "25 delisting(s) from an incomplete crawl". The delistings were real — each page answers 200 with the dealer's own title reading "SOLD 2026 Nissan Patrol Ti Y62 …". What was wrong was the run row they hung off, and the cause was mine.
closeAbandonedRuns() reaps open run rows older than STALE_RUN_HOURS = 2, justified by a comment that measured the thing it was guarding: "no single source takes anywhere near this: the largest budget is ~250 detail fetches at ~9s, about 40 minutes." Fitting the detail budgets to real stock a few hours earlier raised brisbanecars to 8,150 and countrycars to 6,200 — three to four and a half hours of fetching — and the reaper started closing live runs mid-traversal, stamping them finished, incomplete, zero pages. Both went on to finish correctly (countrycars: complete, 2,562 pages, 58 delistings), so the data was never wrong; the run rows were, for a few hours each.
The false alarm was the visible half. The same constant gates anotherCrawlIsRunning(), and a run dismissed as stale stops being a claim on the source — so a second pass could have started against hosts the first was still fetching, doubling the request rate on every shared host. The cron wrapper's pgrep would have caught it locally, but the database claim is there for a crawl started by hand or from another machine, and it had quietly stopped covering the two largest sources.
First fix was to derive the window from the largest budget. It works and it is the wrong trade: at 8,150 pages it computes to nineteen hours, so a crawl that really died would block every scheduled pass for most of a day. That is the failure the window exists to prevent, moved rather than removed.
Age was never the right signal. A crawler writes observations continuously, so whether one is alive is observable rather than inferred from a clock: a run is abandoned only if it is old *and* nothing has been written for twenty minutes. A four-hour brisbanecars run recording cars every few seconds is plainly working; a dead process stops writing within one page fetch.
Liveness is checked corpus-wide, not per source. Per source it would still reap a run whose pages happen to be 404ing for a stretch — that source writes nothing while its process is fine — and reaping a live run is the whole bug. Any recent observation proves a crawler exists, and if one exists its rows are not abandoned. An unreadable check returns "not quiet": unknown must never license reaping.
Verified against the running pass: last observation 0m ago, so nothing would be reaped, and dealer:peterwarren — open 2h+, mid-traversal — would have been reaped by the old rule. It was about to happen a third time.
12:39 am AWST · 5f26de6
From a fourteen-agent pass whose five adversarial verifiers all returned needs-change. Their corrections are in the document rather than the drafts they corrected.
Three measurements decided it. model_ticker('ford','ranger') takes 19.1 seconds against an 8 second cap, so /model is failing now and cannot take charts until it moves off the request path. daily_model_rollup holds two non-contiguous days and the longest contiguous run for all 723 cohorts is 1, while live went 49,673 to 137,764 between them, so any line drawn there is a 177 percent rally made of crawler expansion. And Ford Ranger is 5,820 listings against 2,644 vehicles, so vehicle dedupe is mandatory, after which exactly nine models corpus-wide exceed 800 points and the downsampling debate disappears.
Bespoke SVG stays. Observable Plot lost because it cannot run in a server component at all and because its defaults fabricate. Measured d3-array 9.3, our interaction layer 1.3, against a site that ships 174.9 kB today.
The spine is an evidence ladder: charts declare what they need, a 6-hourly evaluator resolves a rung per cohort, and interaction can only lower it. Resolution is by first failure rather than highest pass, because the intuitive form lets a cohort skip a rung it fails.
The finding that justifies the whole mechanism: the pre-registered Tell's 2,000-cut gate opened days ago at 5,997 cuts and nothing fired, because a gate written in prose is not a gate.
12:37 am AWST · 1cc0530
Two defects, both found by the charting research reading cron.job and the migration record against the tree rather than trusting either.
Migration 0087 rewrote the depth-boards cron command to add refresh_sale_speed_board() and dropped refresh_market_boards() while doing it, because the command is one string replaced whole rather than appended to. The four market boards had been frozen since their last good run and /market was serving figures in the present tense that had stopped moving. They were four hours old when caught, inside the twelve hour limit, so check_stale_boards() was correctly quiet and would have alerted tonight. The check works. It cannot see a job that will never run again, only the staleness that follows, which is an argument for asserting the schedule itself and not only its output.
sale_speed_board arrived in 0087 and was never added to the freshness spec, so it was unasserted. It is in it now.
Separately, 0088's two partial indexes existed in the database but the database held no record of them: they were built live with CREATE INDEX CONCURRENTLY, which cannot run inside a transaction and so could not go through apply_migration. The file sat in the tree describing a change the migration record had never heard of, the mirror of the thirteen migrations that once existed only in the database. Recorded properly rather than deleted, since the change is real and idempotent.
12:17 am AWST · a8bdde5
Records the detail-budget sizing and the www/sitemap fix in COVERAGE-CEILING.md, alongside the open question they leave: fifteen sources whose sitemaps read fine standalone but whose traversal still reports 12 or 24 found. The prediction to check on the next full pass is written down rather than assumed, because their last runs predate the config that should fix them.
12:15 am AWST · 825bf72
Autopact advertises 3,250 vehicle URLs in its sitemap. Every pass for weeks reported `found=12` or `found=24`, and those came from the HTML index — not one sitemap entry survived to be fetched.
crawl.ts compares each sitemap <loc> against `sourceHost`, derived from the configured indexUrl, with `u.hostname !== sourceHost`. Autopact's indexUrl is the apex autopact.com.au; its sitemap lists canonical URLs on www.autopact.com.au. Every entry was dropped on the prefix, before any vehicle test looked at it. The same equality gated child sitemaps, so a www child sitemap was never queued either.
sameSite() discounts a leading www on both sides and nothing else — a sitemap listing another domain is still somebody else's inventory. It lives in sitemap-scan.ts with the other patterns, which exists precisely because a second copy of this logic drifting apart is what cost 97 sources their inventory once already.
Measured on real sitemaps, strict-host against the fix:
autopact 0 -> 3250 pakenhammazda 0 -> 182 (base-url host) mandurahgwmhaval 0 -> 167 (base-url host)
The narrower claim is the correct one. The harness first compared against base_url and reported gains for cherywindsor and dubbohyundai too; crawl.ts:305 derives sourceHost from indexUrl, and against that host both already matched at 136 and 57. Fixing the harness dropped the headline from 3,472 to autopact's 3,250 — the other two are real sources with a real ceiling, but this is not the fix for them, and crediting it with sources it does not touch would have hidden that.
Those remaining sources have a separate ceiling: 15 of them hold 624 live listings between them while their traversal finds 12 or 24 a pass. They are why the delist guard has been busy — each becomes a delisting candidate every pass and is saved only by the 200-response check. Their sitemaps read fine standalone (cherywindsor 136, dubbohyundai 57), so the fault is between the config and the traversal, not in the sitemap. Next thread to pull.
12:02 am AWST · d9ee133
126 sources were capped below their own yards. Between them they found 85,483 listing URLs and could reach 46,200 in a pass; the other 39,283 were discovered and then left for a later pass. That is roughly a quarter of the corpus waiting on a config default.
Deferred, not lost, and the distinction is the whole reason this is a tuning change and not a bug fix. pendingDetail is ordered never-seen first then oldest-seen first, so truncation rotates and every URL is reached eventually; delistings are inferred from presentUrls — the full index traversal — so a short detail budget has never been able to fabricate a sale.
What it costs is freshness. A yard of 3,869 cars read 200 at a time needs twenty passes to be seen once, and at six hours a pass the first observation is five days old before the last car is read at all. Prices move faster than that, and a series assembled from observations five days apart is a worse valuation input than one six hours apart.
Sized against live stock rather than listings_found, because found counts URLs and not all of them are cars. Most sources are honest — brisbanecars finds 1.05 URLs per live listing, countrycars 1.00, booran 0.98 — but pattersoncheney finds 3,872 and holds 468. Sizing that to found would have bought it a 4,850-page budget and spent ~3,400 polite fetches a pass on URLs that have never yielded a car; it gets 700.
Net +33,741 detail pages per pass, about +1.75h at the measured rate. Per-host politeness is untouched: politeFetch still serialises per host and still waits the greater of our delay and robots Crawl-delay, so a bigger budget reads a host for longer, never faster.
Two things found by checking rather than assuming:
- The first formula ratcheted. max(live, current) * 1.4 multiplies the cap it set last time, so a second run wanted 17 more raises after the first wrote 126. Headroom belongs on observed stock; "never lower" is a separate clause. It converges now — a re-run reports zero. - The headline claimed 52,150 by counting budget granted rather than pages fetched. A source only fetches what it found. 33,741 is the honest number and it is the one reported.
source_max_found() aggregates in SQL: crawl_run is well past PostgREST's 1,000-row cap and grows 686 rows a pass, so reducing it client-side would have started silently reading a fraction of the runs. It counts live rows from `listing`, not listing_current — the view resolves two laterals and a correlated event count per row to render a page, none of which answers "how many URLs will the crawler fetch", and going through it exceeded even a 60-second ceiling.