Spending AI reluctantly

The fastest way to kill a production AI system is to let the model do everything. Not because it fails, but because it succeeds at ten times the cost and latency the workload actually needed. The discipline that keeps LLM pipelines alive in production is spending intelligence the way a good engineer spends compute: reluctantly, and only where nothing cheaper works.

Our reference system is Contract Pulse, a prototype we built to normalize awarded contract data from fragmented municipal portals into one sales radar. The problem is a classic extraction workload: some sources publish clean open-data APIs, some publish HTML tables, some publish scanned PDFs. The naive architecture points a strong model at all of it. The disciplined architecture almost never lets the strong model run.

Tier the ingestion before the model sees anything

Data enters through the cheapest tier that works. Sources with structured APIs cost one configuration dictionary and zero model calls. Sources without them get a cached extraction recipe: the pipeline learns the shape of a portal once, then reuses that recipe on every subsequent visit. Only genuinely hostile sources fall through to the last tier, an LLM-driven crawler. The model is the exception handler of the ingestion system, not its default path.

Escalate on confidence, not on faith

Inside the extraction step the same principle repeats. Documents run first through a fast, cheap model with forced structured output, so the response is always a typed record rather than prose that needs parsing. Each extraction carries an overall confidence score. Only when confidence drops below a threshold does the document escalate to a stronger model. Most documents never escalate, which is the entire point: the expensive pass exists for the hard ten percent, and the easy ninety percent rides the cheap one.

Two implementation details do a surprising amount of work here. Prompt caching on the system prompt and schema keeps the bulk pass close to free on repeated structure. And forcing tool use, rather than asking politely for JSON, removes the retry loops that quietly double the bill in most pipelines.

Provenance is a schema field, not a promise

Every AI-extracted value in the system stores four things alongside itself: its confidence, the quoted source snippets that support it, the model that produced it, and the source document URL. The dashboard badges AI-derived fields. When a user questions a number, the answer is one click away, in the source's own words. This is not decoration. It is what makes the system correctable: a wrong extraction can be traced, understood, and fixed at its tier, instead of eroding trust in the whole radar.

The pattern, generalized

Cheapest tier first. Escalation gated on a measured signal, never on vibes. Structure forced at the boundary. Provenance stored with every derived fact. None of this limits what the AI can do; it defines when the AI is worth running. In our experience the pattern cuts inference cost by an order of magnitude while making the output more trustworthy, because every number can explain itself.

The system behind this essay, including the architecture diagram and the honest section on what is still prototype-grade, is documented at /work/contract-pulse.