From six city portals to a quarter-million contract records
- Client:
- Contract Pulse, an ATS product
- Industry:
- Public-sector sales intelligence
- Status:
- In development
- Stack:
- Python · PostgreSQL · Gemini · Claude · React · Google Cloud Run
Awarded-contract data in the US sits in fragmented city, county, state, and school-district portals: some publish clean open-data APIs, some publish HTML tables, some publish scanned PDFs. A sales team that wants to know what a public buyer just bought, from whom, and when the contract expires has no single place to look.
Contract Pulse began as a one-week prototype built to prove an architecture, and the architecture held: the same tiered pipeline now ingests more than 160 sources on a weekly unattended schedule, with a quarter-million contract records normalized behind real authentication. The core idea is unchanged, cost discipline as design: data enters through the cheapest tier that works, AI is spent only where it earns its keep, and every AI-derived field carries its own evidence.
The system
Three decisions that mattered
Ingest through the cheapest tier that works
Sources with open-data APIs cost one configuration dictionary and zero model calls. Sources without them get a cached extraction recipe. Only genuinely hostile sources get the agentic LLM crawler. The pipeline spends intelligence the way a good engineer spends compute: reluctantly, and where nothing cheaper works.
Escalate models on confidence, not on faith
Extraction runs on a fast model with forced structured output and prompt caching. When overall confidence drops below threshold, the document escalates to a stronger model. Most documents never escalate, which is exactly the point: the expensive model is an exception handler, not the default.
Provenance is a schema field, not a promise
Every AI-extracted value stores its confidence, quoted source snippets, the model that produced it, and the source document URL. The dashboard badges AI-derived fields. When a salesperson questions a number, the answer is one click away, in the source's own words.
The AI layer, honestly
Extraction and categorization run on a pluggable fast-model provider, Gemini today with Claude as the automatic fallback, under forced structured output so every response is a typed record. A separate Claude-powered endpoint interprets natural-language search into filters, rules first, AI optional. The confidence gate, the recipe cache, and the evidence store are deterministic scaffolding around the models. This is what we mean by AI with guardrails: the models work inside an architecture that assumes they will sometimes be wrong.
Where it honestly stands
This section used to say the runs were manual, the scheduler missing, and the auth demo-grade, because they were. That label came off in July 2026: ingestion now runs unattended on a weekly schedule as a Cloud Run job, documents live in private object storage behind short-lived signed URLs, and the dashboard sits behind real authentication with invite-gated signup. What is still honest to say: enrichment is partial, roughly half the corpus carries an LLM category so far, and the product serves a pilot deployment rather than paying tenants. The one-week prototype proved the architecture; the months since have been the unglamorous work of making it run without us.
Results, and how they were measured
160+
live public-sector sources
167 active sources counted in the live registry on 2026-08-01, across city, county, state, education, and cooperative portals.
2-tier
model escalation, confidence-gated
A fast bulk pass, with escalation only below the 0.55 threshold.
240k+
contract records normalized and served
249,361 canonical records counted live on 2026-08-01; about half carry an LLM category so far.