Industrial-grade ingestion from hostile sources
- Client:
- ATS infrastructure
- Industry:
- Data engineering
- Status:
- Operating daily
- Stack:
- Python · PostgreSQL · SQLite · Google Cloud Storage · systemd · 2Captcha
Government portals fight automation. CAPTCHAs guard the front door, sessions expire mid-scrape, layouts change without notice, and documents arrive as archives of unpredictable size and quality. The ingestion layer behind TenderPulse treats all of that as a design constraint rather than a surprise.
The result is the least glamorous system we run and the one we are proudest of: a scraping service that has been feeding the data layer on a daily schedule, surviving portal changes, connection drops, and its own restarts, without losing data.
The system
Three decisions that mattered
One canonical engine, thin portal shims
Most of the portals share a common platform underneath their different skins, so one scraping engine drives thirty-seven of them through configuration shims of well under a hundred lines each. A layout fix lands once and thirty-seven portals inherit it. The alternative, dozens of forked scrapers, is how ingestion codebases rot.
The queue is a database, not a file
The upload queue began life as a JSON file and corrupted under concurrent writes. It is now a SQLite write-ahead-log job store feeding a worker pool, and shutdown drains the queue instead of killing it. No data has been lost since the rebuild, and that is a measured claim from the run ledger, not an aspiration.
Every scrape can die safely at any moment
Each source runs in its own process with a time budget, checkpoints, and resumable state, under an orchestrator that escalates termination politely and records every run. Portals will fail; the design assumes it. What matters is that a failed run costs a retry, never a corrupted dataset.
Where the AI is not
There is no AI in this system, and that is the point worth making. Categorization is keyword matching, CAPTCHA solving is a commodity API, and everything else is deliberate systems engineering. The AI lives downstream in TenderPulse, where it has clean, normalized data to work with, because an intelligence layer is only as good as the ingestion underneath it.
What broke, and what we changed
The original JSON queue lost uploads under concurrency; the SQLite WAL rebuild eliminated that class of failure. Workers used to die mid-upload on shutdown; the drain protocol fixed it. Downloaded archives occasionally turned out to be hostile in size; the pipeline now defends against zip bombs. The commit history reads as a hardening timeline, and we treat that history as part of the proof.
Results, and how they were measured
5,500+
new tenders in one fleet run
Out of 18,270 discovered and deduplicated across 39 sources in that run's ledger, July 2026.
0
data lost since the queue rebuild
Measured across the run ledger, through restarts and portal failures.
37
portals on one canonical engine
Via configuration shims; two sources run dedicated engines.