Notes from running large-scale data acquisition in production.

The scraper works on a laptop in month one. Month three is when it dies, and almost never because the site got harder. The four failure modes I see across audits: source choice before architecture, cost-per-document never measured, anti-bot strategy that can't survive a single vendor change, and "scrape now, structure later" pipelines that never get structured. A field guide based on a decade of production systems.

Most teams price scraping at proxy cost. Real cost is proxy + compute + retry amplification + anti-bot vendor + human babysitting + data quality cleanup + replacement when the source breaks. A model for cost-per-acquired-document and where the actual margins live.

Managed scraping vendors are good at exactly the problems they're designed for and quietly terrible at the rest. A decision framework for build vs. buy on proxies, scraping APIs, and full managed data feeds - written from the buyer side, not the vendor side.

"We'll just embed everything" is not an acquisition strategy. Why most AI products plateau on data quality and what a real ingestion pipeline looks like - source selection, extraction tier, structure-first vs. structure-later, refresh cadence, and the part nobody talks about: continuous coverage.

It works on your laptop because your laptop is the easiest case it will ever face. The five differences that kill it in production: datacenter IP reputation, volume-triggered defenses, the long tail of ugly pages, time, and the worst one - the silent failure that returns 200 OK and writes empty rows while every dashboard stays green.