Why most scraping projects die in month three
The scraper works on the laptop. It pulls a few thousand records. Everyone's happy. That's month one.
Month three is when it dies, and almost never because the target site got harder. It dies on decisions nobody wrote down back when the thing looked like it worked. By the time someone calls me, the broken crawler is the symptom, not the disease.
I've seen the same four failures over and over, across a decade of running acquisition at ZoomInfo and now in audits. None of them are exotic. That's the frustrating part. They're all avoidable, and they all stay invisible right up until the month the project falls over.
They picked the source before they picked the architecture
It starts in a meeting. Someone says "we need LinkedIn data" or "we need every law firm in the country," and an engineer goes off to scrape that one specific thing. The source got picked before anyone asked what shape the system around it needs to be.
Then the requirements show up that were always going to show up. Refresh it weekly. Dedup against what we already have. Handle the company that changed its name. Track when a field goes from populated to empty, because that turns out to be the signal the product actually sells.
A scraper isn't an architecture. The architecture is the scheduling, the dedup keys, the change detection, the way you store raw payloads so you can re-parse them when the schema shifts. If you build the scraper first and bolt those on later, you rebuild the scraper. I've watched teams do this three times for the same source before someone asks why.
Decide how you'll know a record changed before you write a single line that fetches one.
Nobody measured cost per document
Ask a team what it costs them to acquire one record and you usually get a blank look, then a number that's only the proxy bill. The proxy bill is maybe a third of the real cost. (I wrote a whole separate log on this, because it's where most of the money quietly leaks.)
The reason this kills projects: without a cost-per-document number, you can't tell whether the project is succeeding. A crawl that works but costs $4 a record to maintain is a failure dressed as a success. You won't find out until the cloud bill arrives and someone senior asks what the line item is. Then the project gets cancelled for being expensive, when really it was just never measured.
The anti-bot strategy was one trick
This is the technical failure people expect, and it's real, but it's almost never about cleverness. It's about fragility.
A team finds one technique that works on a hard target. A specific header set. A particular proxy vendor. A browser automation trick that slips past detection today. They ship it. It works. And the entire pipeline now depends on a single thing that the target site, or the vendor, can change without warning.
The fix is to never bet on one technique. The systems I build run a three-tier fallback: a cheap fast path first, then a heavier path, then a full stealth browser with residential rotation only when the first two fail. The cheap path handles most traffic. The expensive path is there so a target hardening its defenses is a cost increase, not an outage. That's the difference between "our crawler is down" and "our crawler got 8% more expensive this week." One is a fire drill. The other is a Tuesday.
If your reliability depends on one trick, you don't have a strategy. You have a countdown.
"Scrape now, structure later" structures never
This is the one that looks fine the longest. The team fetches raw HTML, dumps it somewhere, and tells themselves they'll extract clean fields when they need them. Acquisition feels done. Everyone moves on.
Six months later there are two terabytes of raw pages and no clean dataset, because "later" is always lower priority than whatever's on fire today. The extraction work that should have been part of the pipeline becomes a project of its own that nobody wants to staff. The data exists and is simultaneously useless. For an AI team this is fatal, because the model needs structured, current input, not a landfill of HTML from last spring.
Structure-first is more work up front and it's the only version that ships. You don't have to extract everything. You do have to extract enough that the output is usable the day it lands, and you have to keep the raw payload so you can go back when requirements grow.
The pattern under the pattern
Every one of these is the same mistake wearing a different coat: treating acquisition as a script you write once instead of a system you operate. Scripts are cheap and they rot. Systems cost more up front and they compound.
The good news is that the failures are predictable, which means they're catchable in week one instead of month three, before you've spent two quarters and a headcount finding out the hard way. That's most of what an audit is. I look at the four things above, in that order, and tell you which one is going to bite you. Usually you already suspect which one. You just haven't had a reason to say it out loud.