Every AI intelligence product is standing on an ingestion layer, and most of them are standing on a cron job someone wrote in an afternoon. This essay is about what the ingestion layer looks like when you treat it as infrastructure: the unglamorous engineering that decides whether the intelligent system above it gets clean data every day or quietly starves.
Our reference point is the pipeline behind TenderPulse, our procurement intelligence product. It ingests government tender portals daily. Government portals are close to the worst case for automation: CAPTCHAs at the front door, sessions that expire mid-scrape, layouts that change without notice, and documents that arrive as archives of unpredictable size and content. Nothing about them is stable, and the pipeline has to be.
Assume every scrape dies. Design for the retry.
The first structural decision is process isolation. Each source runs in its own process with a time budget, under an orchestrator that records every run in a ledger and escalates termination politely: a graceful signal first, a hard kill only after a grace period. One portal hanging must never take the night's other runs with it.
The second is resumability. Long scrapes checkpoint as they go, so a run that dies at item eight hundred restarts at item eight hundred, not at zero. Combined with time budgets, this changes the economics of failure completely: a crash costs a retry, never a lost day and never a corrupted dataset. Once failure is cheap, you stop being afraid of it, and the whole system gets simpler.
The queue is a database, not a file
The most instructive failure in this system's history was self-inflicted. The document upload queue began life as a JSON file. It worked in development, survived early production, and then corrupted under concurrent writes, exactly as any database engineer would have predicted. The replacement is a SQLite job store in write-ahead-log mode, feeding a worker pool, with one more rule that matters: on shutdown, workers drain their queue instead of dying mid-upload. No data has been lost since the rebuild, and that is a measured claim from the run ledger, not a hope.
The general lesson: any file that two processes write is a database schema you have not admitted to yet. Admit it early. SQLite costs nothing and removes an entire class of failure.
One engine, thin shims
Several of our target portals turned out to share a common platform underneath different skins. So one canonical scraping engine drives them all, through configuration shims of well under a hundred lines each. When a layout changes, the fix lands once and every sibling portal inherits it. The tempting alternative, forking the scraper per source, is how ingestion codebases rot: five forks drift apart, and within a year you are maintaining five subtly different bugs.
Where the AI is not
There is no AI in this layer, and that is deliberate. Categorization is keyword matching. CAPTCHA solving is a commodity API. Retry logic is arithmetic. The AI lives downstream, where it summarizes and scores tenders against clean, normalized, deduplicated data. An intelligence layer is only as good as the ingestion underneath it, and putting a model inside the ingestion loop would have added cost and nondeterminism exactly where the system needs neither.
If you are building on scraped or partner-fed data, the questions that matter are boring on purpose. What happens when a source hangs. What happens when the process dies halfway. What happens when two workers write at once. What does the run ledger say happened last night. Answer those in the architecture and the intelligent system above gets to be intelligent. The full case study, with the architecture diagram, is at /work/tender-scraper.
Talk this through with the author
A Systems Strategy Session is thirty minutes on the architecture of your operation, with the engineer who wrote this.