Know what is not a car: caravan makes are out of scope, not unrecognised
11:48 pm AWST · f81fc81
Dealers list caravans and motorhomes beside cars, so Jayco and Adria arrived
through the same pipe and sat in the review queue as though nobody knew what
they were. A queue that fills with things already decided hides the makes
that genuinely need a decision, so recognised-but-out-of-scope is now its own
state and recanonicalise applies it.
Aion and GAC — Chinese marques now selling new here — were genuinely missing
and are added. The queue is down to one entry, and it is the right one: a
dealer publishing 'Omoda Jaecoo' as a make has named two Chery sub-brands at
once, and picking one would be inventing the answer.
Market by segment: what is actually on the lot, with running cost and safety
11:46 pm AWST · 0ee06d2
The enrichment layer was queryable but invisible. The market page now leads
with a segment table — live count, median ask, median odometer, median age,
average fuel consumption and ANCAP stars — which is the overview the corpus
was built to support, and the first view that answers a question about the
market rather than about one car.
Every row also says what share of it was classified by the Green Vehicle
Guide rather than by parsing the dealer's body text, because that is the
difference between a government classification and a guess, and it is exactly
where the wagon/SUV confusion lives.
Health check: one line when fine, ALERT lines when not
11:45 pm AWST · 945d53e
Written for unattended running on a timer. The quiet case is a single line so
a night of checks stays readable; anything that would put an unreal number on
screen — a price change spanning index and detail, a delisting from a partial
traversal, an unlabelled observation, an implausible price, or a crawl that
has gone quiet while still claiming to run — prints an ALERT and exits
non-zero.
Run enrichment and the integrity checks after every scheduled crawl
11:44 pm AWST · 074b780
Enrichment only processes vehicles that lack it, so it stays cheap and keeps
the context layer level with a corpus that grows every few hours. The
integrity checks then fail the workflow if the crawl produced something that
cannot be true — a price change spanning index and detail, a delisting from a
partial traversal. A red workflow is much cheaper than a wrong number on
screen, and nobody is watching at 4am.
Enrichment: resolve every vehicle against the public datasets once, not per query
11:44 pm AWST · 0046c07
The context layer was joined by fuzzy name match at query time, on the
valuation screen only. That made the depth invisible everywhere else and
impossible to aggregate — there was no way to ask what the SUVs listed in WA
average on fuel. Each vehicle is now resolved once into vehicle_enrichment
(segment, fuel consumption, CO2, annual fuel cost, ANCAP stars, national parc
for the make, recall count), and listing_enriched exposes it alongside every
listing so the whole corpus can be queried through it.
Across 4,727 vehicles: segment 99%, fuel/CO2 80%, ANCAP 83%, national parc
87%. Every link records how it was matched — make+model is a weaker claim
than a match agreeing on year and variant, and a screen showing fuel
consumption should be able to tell them apart.
Segment needed the datasets to disagree to get right. Australian dealers
follow the Redbook convention of calling an SUV a "Wagon", so classifying
from the listing's own body string produced a 1,979-strong "Wagon" segment
that was in fact RAV4s, Foresters, Tucsons, MU-Xs and Model Ys — it would
have put a plainly wrong picture of the market on screen. The Green Vehicle
Guide classifies those same cars as SUV, so it decides where a match exists
and the listing string is the fallback. SUV is now the largest segment at
1,881, which is what the Australian market actually looks like, and
segment_source records which dataset decided each one.
are closed at the start of every crawl. A laptop sleeping mid-pass would
otherwise leave rows that block the scheduled GitHub Actions crawls for the
rest of the night — exactly when unattended collection matters most.
Corpus integrity checks, verified to actually catch what they guard against
11:37 pm AWST · b7bd390
Fabricated price changes reached the database twice tonight, so the invariant
that would have caught them is now asserted rather than assumed:
- no price change spans an index/detail boundary (the actual bug)
- every present observation records which view produced it
- no delisting inferred from a crawl that did not traverse the inventory
- no implausible prices, no VIN mapping to two vehicles, no stale live listing
The checks that need joins live in SQL, against the database rather than in
application code that may not run. Each was tested by feeding it the failure
it guards against inside a rolled-back transaction: the cross-view check
returns 1 for a synthetic stale-index-against-fresh-detail event and 0 for the
real corpus, so it is not a check that merely passes today.
Six price changes were fabricated by my own code; fix the cause and delete them
11:19 pm AWST · c1627ee
The corpus produced its first six price changes tonight. All six were false.
An inventory index and a listing's own page do not agree. On Dealer Studio
the index is an Algolia search index and it lags: a 2022 Kia Carnival whose
detail page recorded last_price_change_at 02:14Z and served $31,990 was still
being served as $32,990 by the index thirteen hours later. Both reads were
honest; they were of different things. I had added index crawling for cheap
re-pricing without making the diff aware of it, so a fresh detail read against
a stale index read looked like a dealer raising prices. Confirmed by fetching
the detail page live: still $31,990, unchanged.
That is precisely the fabrication this product must never produce, and it
would have been on screen at the demo as the headline differentiator.
Fixes:
- Every observation records the view that produced it, index or detail.
- Price changes are only ever diffed within one view, so each is its own
series and a real index-to-index change still registers.
- listing_current prefers the latest detail observation over a newer index
one, because the listing page is what a buyer sees and the index is a
cache that can be hours behind.
- The six events are deleted, verified against live detail pages first.
Value the actual variant: weight comparables on drivetrain, transmission and fuel
11:14 pm AWST · 7b798ea
The engine matched on make, model and year and ignored spec that was already
captured well — transmission on 99.5% of vehicles, body 99.2%, fuel 96.2%,
drivetrain 83.5%. A Ranger XL 4x2 manual and a Ranger Wildtrak 4x4 auto share
a make and model and sit $26k apart in our own corpus, and they were weighted
identically. The same 2019 Ranger at 95,000km now values $37,254 as a 4x4
against $34,217 as a rear-drive, which the engine previously could not say.
The picker gained an optional refine row for those three, shown only once a
model is chosen and listing only values present in the corpus.
Building it surfaced the taxonomy problem applied to specs: one Ranger cohort
spelled its drivetrain five different ways, which split the options a user
picks between and made a differently-spelled comparable look like a different
car. Specs are now canonicalised for matching and display — five drivetrain
options collapse to two, eight transmissions to two — and matching 4x4 draws
on an effective sample of 70.7 rather than 48.9. Raw spellings stay on the
vehicle row and in the captures.
Two crawl cadences: a deep pass for coverage, a fast pass for price movement
10:53 pm AWST · 636ad2f
Breadth and freshness want opposite things. A deep pass walks every sitemap
at one fetch per car — that is how coverage grows, and it is the only pass
that can infer a delisting, since absence only means something when the whole
inventory was seen. But at 156 sources it takes hours, and price changes come
from frequency, not depth.
--fast crawls only the inventory index pages, where a single fetch yields
around twenty cars *with their prices*, so the corpus can be re-priced in
minutes. It never marks delistings. Deep runs twice daily, fast every three
hours, and the in-flight guard applies to both: per-host politeness is
enforced per process, so overlapping passes would hit shared hosts twice as
often. They are scheduled apart rather than exempted.
Also seeds 92 more verified dealer sites from the second discovery sweep:
64 -> 156 sources, across all states including TAS and NT.
Coverage: distinguish 'not yet crawled' from 'crawled and found nothing'
10:51 pm AWST · e2ee515
A source added since the last pass rendered identically to one that was
visited and yielded zero, which reads as broken extraction when it is just a
queue. The header now splits configured / crawled / contributing, and the
last-crawl column says 'not yet crawled' rather than 'never'.
Stop mixing drive-away and ex-on-roads prices in one range
10:50 pm AWST · e800bbd
An Australian ad states either a drive-away price — rego, stamp duty and
on-roads included — or one excluding government charges, and on a k car
the gap is thousands. The corpus splits almost evenly (1,314 drive-away
averaging $41,094 against 1,257 excluding, averaging $38,114) and the
engine was averaging across both, which compares different numbers and is
the first thing a dealer would catch.
The range is now computed on whichever basis dominates the comparable set,
cars advertised on the other basis are down-weighted rather than dropped
(x0.60, or x0.85 when the ad does not say), and the output states which basis
it is on and how many comparables share it. Deliberately no conversion
between the two: that needs state-by-state on-road costs we have not
ingested, and estimating them would be inventing the number.
Comparables table and API responses now carry the basis per car.
Raise cross-host crawl concurrency 4 -> 8
10:42 pm AWST · adf8102
Politeness is enforced per host — one request in flight, 8s+ apart,
Crawl-delay honoured — and all 64 sources are different hosts, so running
more at once adds no load to any individual site. It only stops the crawler
idling through another host's delay, which roughly halves a full pass.
Cascading make → model → badge → year picker, and stop under-reporting the corpus
10:39 pm AWST · ad02a5d
The Market table was capped at 14 makes, which read as 'this corpus holds 14
brands' when it actually holds 57 makes and 362 models. That is a claim about
coverage made by a display default, so the cap is gone and the heading states
the real totals. Both market aggregates now page through PostgREST's 1,000-row
response cap as well, which was silently describing whichever slice sorted
first.
The valuation form is now a cascade: pick a make, then a model, then badge and
year, then type an odometer. Every option is a value that exists in the corpus
with its live count beside it, so the form cannot ask a question the data
cannot answer, and the counts show how much evidence is behind a choice before
committing to it. Selecting a make clears what depends on it.
Fix CI: generate Next route types before typechecking
10:27 pm AWST · bc57001
CI failed on every push since the web-build job was added: LayoutProps and
PageProps are route types Next generates during dev, build or typegen, so
they exist locally but not in a clean checkout, and tsc ran before the build
that would have created them. Reproduced locally by deleting .next/types.
Recover a whole source: sitemap listings no longer depend on extractor link recognition
10:26 pm AWST · 5fb48bd
Westside Auto reported found=0 against a sitemap holding 1,794 cars — the
largest single source in the corpus, silently contributing nothing. The
orchestrator handed sitemap entries to the extractor as synthetic anchors,
which works for extractors that parse links but not for ones that only
recognise their own page markup, and the nextjs-embedded extractor reads
__NEXT_DATA__.
The extractor is still asked first, since it filters its own platform's URL
shapes best; when it recognises none, same-host entries that look like a
vehicle page are used instead. Verified: 1,794 locs -> 1,794 listing URLs,
and a detail page extracts to a 2020 Mercedes-Benz G with price, odometer
and VIN.
Taxonomy: add Fuso, resolve 28 more vehicles; demo script updated
10:23 pm AWST · 5f28a48
One make stays in the review queue deliberately: a dealer listing 'Omoda
Jaecoo' as its make is two Chery sub-brands in one field, and picking one
would be inventing data. That is what the queue is for.
Demo script rewritten around what is actually on screen, including a section
naming what is genuinely not there yet so it gets said before anyone finds it.
Operations: document the two crawl schedules and the in-flight guard
10:21 pm AWST · 3fa0246
Move the crawl schedule to GitHub Actions so it survives the laptop sleeping
10:20 pm AWST · eade4fd
The observation layer only compounds if the crawl actually runs, and this
machine sleeps a minute after the session ends — which would mean no second
pass overnight and no price changes to show. The schedule now runs on
GitHub Actions every six hours, with a concurrency group and a 5.5h timeout.
The local cron entry stays as a second path: whichever starts first takes the
run, because crawl.ts refuses to start while another crawl is in flight.
A daily job also refreshes the datasets that actually move (fuel, cash rate,
new-vehicle topline); each records its own block rather than failing the run.
Repo is private, so the service-role key lives in Actions secrets.
Refuse to start a second concurrent crawl
10:18 pm AWST · 8ddd822
Per-host politeness — one request in flight, 8s+ gap, Crawl-delay honoured —
is enforced inside a process. Two crawls running at once therefore double the
request rate against every host they share, which is exactly the thing the
crawling rules forbid. The cron wrapper only checked for a local process,
which does not cover a hand-started run or another machine, so the claim is
now checked in the database every crawler shares. Runs older than 8 hours are
treated as dead rather than blocking forever, and --force overrides
deliberately.
the run already in flight.
Confine crawler to source hosts and cap response bodies
10:16 pm AWST · 1233460
Two findings from the SSRF review, both confirmed by independent verifiers:
- The generic jsonld extractor queued URLs straight from JSON-LD 'url'
fields and rel=next without a host check, while every platform-specific
extractor already had one. JSON-LD is content from a page we do not
control, so a crawled page could point the crawler at any host — one whose
robots.txt we never read and whose terms we never agreed to — and spend our
politeness budget there. The private-address guard stops internal targets;
this stops off-host public ones.
- politeFetch buffered whole responses with no byte cap. The 30s timeout
bounds how long a transfer may take, not how large it may be, so a fast
server could stream unbounded bytes into memory. Bodies now stream against
a 25MB cap and the stream is cancelled the moment it is exceeded — robots
.txt reads included.
That closes all 8 findings the 29-agent review confirmed. Nothing survived
verification against the frontend (React escaping holds for crawled strings)
or against secrets handling.
Close the public /value write-amplification path and share one input validator
10:04 pm AWST · ae66fd5
The API validated its body while /value took the same subject from a query
string unvalidated and persisted it verbatim — so an 8KB badge padded into
the URL was stored on every GET, unauthenticated and unthrottled, which is a
way around the limits the API had just gained. Both now use one validator in
valuation-input.ts, and an invalid value is shown to the user rather than
silently ignored.
Valuations are additionally deduplicated: an identical subject and model
version inside an hour reuses its stored record. Four repeat page views now
write one row, and comparable payloads are ~9KB rather than hundreds. The
audit trail SPEC §2 requires is preserved — one row per distinct question.
Fix three more review findings: unbounded valuation writes, /market rescans, probe output
10:01 pm AWST · 964fcbf
- Every /value view persisted the full comparable set as jsonb — up to 400
objects each carrying prose adjustments, hundreds of KB per unauthenticated
request with no retention. SPEC requires storing the working, so the
heaviest-weighted 50 comps are kept (what the range actually rests on and
what a reviewer needs) with the true total recorded in evidence.
- /market recomputed corpus-wide aggregates on every request: ~65 database
round trips, cache explicitly disabled, drivable by anyone. Aggregates
change only when a crawl writes, so they are memoised for 60s with
concurrent callers sharing one in-flight computation and a stale value
surviving a database blip. 0.52s -> 0.10s.
- Extractor probe modes write live-fetched third-party pages to
.dealer-studio-probes/, which no ignore rule covered — one routine
'git add -A' would have committed them. Now ignored.
Fix four API defects found by security review, verified by attacking them
09:59 pm AWST · 7da2e38
- ilike takes a *pattern*, and make/model went in unescaped: a caller
sending '%' matched the entire corpus, producing a confident-looking
valuation built from unrelated cars and scanning every row. All ilike
call sites now escape the value; a wildcard returns an honest zero.
- consume_api_quota was documented as atomic and was not — count and insert
were separate statements with no lock, so concurrent requests for one key
both passed the check. The key row is now taken FOR UPDATE. The claim in
OPERATIONS.md is corrected rather than left overstating the guarantee.
- Rejected requests were written with status 429 and then counted toward
usage, so a retry loop against a spent quota locked the key out well past
the rolling window. Only served requests count now.
- The JSON body was cast, not validated: a non-string make or non-numeric
year surfaced as a 500 or persisted a NaN valuation. Fields are now
type- and range-checked and return 400 with the specific reason.
Operations: test suites, re-parse loop, taxonomy queue, seven datasets joined
09:56 pm AWST · 166b027
API documentation page: the socket, made visible
09:55 pm AWST · 6920e80
Documents /api/v1/value with a real request, the full evidence-carrying
response shape, and the ok:false insufficient-comparables reply — which is a
legitimate answer, not an error, and callers should treat it that way.
Fix nine SSRF bypasses in the private-address guard, found by testing it
09:54 pm AWST · e438272
The guard matched on the *text* of an address, and an address has many legal
spellings. Confirmed bypasses, all now blocked and covered by
scripts/test-ssrf.ts (45 cases):
- http://[::1]/ was allowed outright: URL.hostname keeps the brackets on an
IPv6 literal, so isIP('[::1]') is 0 and the private check never ran.
- Expanded spellings evaded the prefix strings entirely — 0:0:0:0:0:0:0:1 is
loopback but is not the literal '::1'.
- Loopback and 169.254.169.254 smuggled inside v4-mapped, NAT64 (64:ff9b::/96)
and 6to4 (2002::/16) wrappers were treated as public.
Addresses are now parsed to bytes and range-checked, with every embedded-IPv4
form unwrapped and re-checked, plus the ranges the original missed (6to4
relay anycast, TEST-NET-2/3, discard, documentation). Per-host politeness
queues key off the bare hostname so an IPv6 host is rate-limited as one host.
CI now runs the SSRF and statistics suites — both are pure functions needing
no network or database, and both guard properties that fail silently.
Content-address raw captures, and prove the re-parse loop works
09:50 pm AWST · c09a9a1
Re-observation re-fetches unchanged pages by design, and at ~200KB each that
would have multiplied into gigabytes of byte-identical bodies (274MB after
one partial pass; ~3.4GB per full pass at 64 sources). Bodies are now stored
once per distinct sha256: a repeat capture records the observation with a
null body and resolves the bytes by sha. A page that genuinely changed has a
different sha and is stored in full, which is exactly the version history the
observation layer is for.
raw.ts is the single read path — a reader that went to body_gzip directly
would see deduplicated captures as empty. reparse.ts closes the loop SPEC
calls the highest-leverage ingest decision: verified re-parsing 8/8 stored
captures with the current extractor and no re-crawling.
Pin Vercel functions to Sydney: /market 2.74s -> 0.44s
09:48 pm AWST · eabd273
Functions were defaulting to Washington DC while Supabase sits in
ap-southeast-2, so every query crossed the Pacific and pages that run several
queries paid for it repeatedly. Same region as the database now.
Four more sources from the second discovery sweep: 64 dealer sources
09:46 pm AWST · da8b5be
Peter Warren was recorded as an un-crawlable marketplace during the survey;
the sweep confirmed it is a Dealer Studio site whose robots permits the
inventory, so it moves in-scope as a dealer source and the stale marketplace
row is retired with a note explaining why.
34 more verified dealer sources: 26 -> 60 across all six states
09:45 pm AWST · 6dcb8ed
Platform-discovery pass found and individually verified 34 additional AU
dealer sites on platforms we already parse — robots.txt fetched, inventory
path confirmed permitted, platform fingerprint and a real listing confirmed
on each page. Sites returning a WAF challenge were excluded, not worked
around. Includes John Hughes, Melbourne City Toyota, Frizelle Sunshine and
Tony White Group.
A source can now carry several index entry points. Dealer Studio's
/search/used-cars yields ~21 fully-parsed listings for a single fetch, while
its sitemap costs one fetch per car — crawling both means a truncated run
still refreshes prices instead of only discovering URLs, which is what
price-change detection actually needs.
Listing detail page: the observation log made visible
09:42 pm AWST · c5413b9
Per-listing view showing full captured specification (VIN, build/compliance
dates, drivetrain, colours), the observed price line, the change timeline,
and the complete append-only observation log with parser version per read.
This is where 'we watch the same car over time' stops being a claim and
becomes something you can look at. Comparables in a valuation now link here
rather than off-site.
Coverage extraction rate now measures the parser against listings actually
attempted; listings found beyond a run's detail budget are backlog, not
parser failure, and were wrongly dragging the rate down.
Close the taxonomy review loop: 103 vehicles resolved without re-crawling
09:37 pm AWST · 5a891ea
Every make added here was surfaced by the review queue against real listings
rather than guessed up front — Geely (78 vehicles), GMSV, Holden Special
Vehicles, Ford Performance Vehicles, Hino, Harley-Davidson, and 'Skoda',
which never matched the ASCII canonical form. recanonicalise.ts re-runs the
taxonomy over stored rows, which is the other half of the queue's purpose:
improving the mapping repairs history instead of only helping future crawls.
Review queue now 2 vehicles / 4 aliases, from 94 / 25.
Insight engine with real evidence bars — and it currently, correctly, reports nothing
09:34 pm AWST · 4ae0dd0
Adds SPEC §3.4: four hypotheses tested continuously, each gated on group
size, a Welch t-test at p<0.05, and a commercially meaningful effect size.
Statistics live in stats.ts and are verified against published t critical
values (scripts/test-stats.ts) — an insight engine with wrong p-values
manufactures confident nonsense.
Two bugs caught by testing against the real corpus rather than trusting the
first plausible output:
- the corpus query silently capped at PostgREST's 1,000-row default, biasing
every insight toward whichever sources sorted first; now paginated.
- cohorts keyed on make+model alone reported a '60% interstate price gap'
that was entirely fleet composition — states stocking older, higher-km
examples of the same model. Cohorts now fit their own price-by-age-and-
odometer curve and compare residuals; the true adjusted spread is ~8% at
p=0.43, so the engine correctly surfaces nothing and says what it tested.
Listing state gains a provenance column: inferred-from-dealer-location is
recorded distinctly from stated-on-the-listing, and the crawler applies the
same rule going forward.
Demo script: the five-minute walkthrough and the CarCrunch positioning
09:04 pm AWST · db970ba
FCAI monthly topline ingest: July 2026 = 103,656 new vehicles
09:03 pm AWST · 247c196
Seventh dataset joined. Only the free machine-readable number is taken;
make/segment detail is paid VFACTS and stays out of scope.
Valuation and extractor quality fixes from first real-data pass
09:00 pm AWST · 502f9cb
Odometer adjustment capped at 25% of the comp's ask and distant-km comps
down-weighted — a 2017 Ranger at 277,000 km no longer gets shifted +$22k to
match a 95,000 km subject. EasyCars dealer-name heuristic no longer promotes
pagination titles ('Page 2') as dealer names (stored rows repaired). Generic
jsonld discover skips action sub-pages (book-an-appointment etc) that robots
rightly disallows — Jarvis extraction errors were all of that class.
Operations: cron cadence, sitemap coverage mechanics, ingest outcomes
08:57 pm AWST · b40a381
Fix: coverage page was gitignored (never deployed); partial build dates crashed inserts
08:55 pm AWST · b0d015e
Root .gitignore's unanchored coverage/ pattern matched web/src/app/coverage/,
so the coverage page existed only locally — anchored to /coverage/ and the
page is now tracked. Auto Home publishes month-precision build dates
(2017-07) which Postgres rejects; dates are now floored to day precision
instead of dropped, recovering the 29 listings that failed there. Coverage
page also gains a marketplaces section with recorded block evidence and a
running-crawl indicator.
Re-observation cadence: cron wrapper crawling every 6 hours
08:51 pm AWST · d0732c0
Skips if a crawl is already in flight. FuelWatch WA daily refresh alongside.
Crontab entries installed on this machine (documented in OPERATIONS.md).
Crawler: sitemap traversal, unseen-first detail rotation, presence-safe delisting
08:50 pm AWST · a7ee273
Dealer Studio, iMotor and Algolia-backed Next.js sites render only their
first page server-side; their XML sitemaps list the full inventory, so the
orchestrator now walks sitemaps (including same-host sitemap indexes) and
filters locs through each extractor's own link recognition. Detail budgets
requires a complete traversal AND absence from the traversed index/sitemap,
so a listing beyond today's detail budget can never be falsely delisted.
Somerton Car Sales disabled: CloudFront 403s our honest UA (recorded).
Platform extractors for all six surveyed dealer platforms, fixture-tested
08:46 pm AWST · 6b3d125
dealer-studio (231 listings across 11 fixtures, price+odo 99.6%), easycars
(slug+card parsing incl. drive-away price type), imotor (both SSR and
embedded-JSON flavours, VIN 100%), nextjs-embedded (__NEXT_DATA__ tree walk,
VIN 100%, build/compliance dates 91%), cox-radius and easylist one-offs.
Registry wired; jsonld remains the generic fallback. Extraction-rate test
scripts under scripts/test-extractors report per-field percentages from the
saved survey fixtures.
Repo agent guide: binding rules and layout for future sessions
08:41 pm AWST · 54d9e69
CI: typecheck and build the web app on every push
08:37 pm AWST · a3e1e63
Placeholder Supabase env so next build succeeds without real credentials;
all pages are dynamic so no data is fetched at build time.
Favicon: the lot mark on ink
08:34 pm AWST · c05aa62
Operations runbook
08:34 pm AWST · f2030b3
Versioned valuation API: /api/v1/value with hashed keys and per-key daily quota
08:32 pm AWST · e32ce7a
Bearer auth against sha256-stored keys, atomic rolling-24h quota via a
consume_api_quota() SQL function, full evidence in every response (range,
confidence subscores, comparables, adjustments, days to turn, context).
issue-api-key script prints plaintext exactly once.
Valuation engine and market movement layer, wired into the surface
08:31 pm AWST · 610d4a6
valueCar(): weighted comparables (year decay, badge, state), Theil-Sen
odometer slope bounded to plausible depreciation, weighted quantile range,
three legible confidence subscores, days-to-turn from observed delistings
only, buy price with visible model parameters. Refuses below 3 comparables
with the real count. Every valuation persists its full evidence per SPEC §2.
Market page renders observed movement only; value page shows range, band,
scatter (yellow subject, grey comps), comparables table, adjustments, and
the public-dataset depth panel.
Survey results, fixtures, context schema, and 44 seeded sources
08:26 pm AWST · 568c827
Full source survey saved to docs/research: 9/9 public datasets verified with
real fetched data, 29 dealer sites verified crawlable (robots checked each),
4 marketplaces recorded as WAF-blocked and out of scope. Survey HTML cached
as gzipped fixtures for extractor development. Context tables (ANCAP, BITRE
fleet, recalls, econ series, region stats, holidays, GVG) migrated. Source
registry seeded: 26 crawlable dealer sites across 5 states, 9 gov datasets,
blocked marketplaces recorded with evidence.
Surface v1: brand system and dashboard with live, honest coverage counts
08:15 pm AWST · 1563b85
Dark-first token set from BRAND.md (yellow only ever carries black), Inter
Tight display + JetBrains Mono tabular figures. Home, coverage, market and
value pages all render real database counts with explicit empty states —
nothing seeded or implied. force-dynamic so counts are fresh per request.
Ingestion core: SSRF-guarded polite fetcher, append-only observer, JSON-LD extractor
08:10 pm AWST · 6de5190
politeFetch validates every resolved address at connection time (closes DNS
rebinding), honours robots.txt per origin including crawl-delay, serialises
per host with an 8s+ gap, and never retries a block. The store resolves
vehicles (VIN first, source-scoped fingerprint otherwise), appends
observations, diffs consecutive observations into price/photo/description/
odometer events, and marks delistings only after a complete index traversal.
crawl_run records extraction rate per source. Generic schema.org JSON-LD
extractor as the platform-agnostic baseline.
Scaffold Next.js app and land the core schema
08:05 pm AWST · 6316a3c
Supabase project mtwpnxhuzolkngutbyaf (ap-southeast-2) created and the four
foundation migrations applied: sources + raw captures, vehicle/listing/
observation separation with append-only observations, taxonomy alias queue +
valuation-with-evidence, and the listing_current derived view. RLS enabled on
every table with anon/authenticated revoked; only the service role reads.
Build spec: what an unsupervised session needs decided in advance
07:52 pm AWST · 8d7dec2
Written for a long-running session that cannot ask questions, so everything it
would otherwise guess is decided here.
The architectural rule first: stay demoable from the first hour and deepen
continuously. "Take as long as it needs" and "there is a meeting Wednesday"
only coexist if the product never enters a state where it does not run. No
big-bang integration, additive migrations only, and every abandoned phase
leaves something that still works.
Data-rich is defined concretely rather than aspirationally: capture every
field a listing carries and not just the four a valuation needs, keep the raw
payload so improved parsers can re-read old captures without re-crawling,
observe longitudinally, and join public Australian context data. The Green
Vehicle Guide, ABS Motor Vehicle Census, ANCAP, recalls, fuel prices and RBA
rates are all free and structured, and they are the difference between a
scraper with a scatter plot and something that can argue its valuation.
Every source is marked with how confident I am that it exists in the form
described, with instructions to verify before depending on it. An unsupervised
agent that assumes a dataset exists will otherwise waste hours or invent one.
Also records that the wow is a true non-obvious insight, not volume, and that
the CarCrunch comparison should be invited and won on depth rather than
breadth.
CI: secret scanning from the first commit
07:47 pm AWST · 1082a62
The README says secret scanning is part of the posture from commit one, so it
should be true rather than aspirational. Same gitleaks job and pin as the other
repos.
Cheap now: this repo will hold Supabase service-role keys and per-source
crawler credentials soon enough, and retrofitting a scanner after a key has
already been committed does not help.
Spotlot: brand direction and project brief
07:46 pm AWST · 3084b87
Australian car-market intelligence and trade-in valuation. Standalone product
that also plugs into Dealerloop, with Dealerloop as customer zero rather than
the owner.
No code yet. This commit is the ground rules, so the build session starts from
decisions rather than assumptions:
- BRAND.md, direction taken from the reference Taj chose, with colours sampled
from the image rather than estimated. The load-bearing rule is that yellow is
a fill and never an ink: on white it is about 1.1:1 and unreadable, so a price
rendered that way is invisible. It only ever carries black.
- brand-spec.html, the same thing rendered, including the failing combination
shown rather than described.
- README, covering the separation rules, the crawling constraints, and the
security posture to start with rather than retrofit.
- A note in Brand Inspiration making clear that the reference is someone else's
work, kept for direction and never to be published or presented as ours.