Crawl economics: what a page actually costs you

Almost everyone prices scraping wrong, and they all make the same mistake: they price the proxy and call it the cost. Then they're surprised when the project's real cost is three to five times that, and surprised again when the cheap source turns out to be the expensive one.

Cost per acquired document is the only number that tells you whether your acquisition is healthy. Not pages fetched. Not requests sent. Cost per good record that lands in the place your product reads from. Here's everything that goes into it, in roughly the order people forget about them.

Proxies are the part you already counted

This is the visible cost, so I'll keep it short. Residential and mobile proxies are billed by bandwidth or by request, and they're priced to feel small per unit. They are small per unit. They're also the only line item most teams track, which is exactly why the total always comes in over budget.

Note the number, then keep going, because everything below is the part that wasn't on the invoice.

Retry amplification is the silent multiplier

Here's the one that gets people. You quote yourself "we need one million records, at this proxy rate, so it costs X." But you don't fetch a million pages to get a million records. On a hard target you fetch a million, half fail the first time, you retry, some fail again, you retry again. By the time you have a million clean records you might have made two and a half million requests.

Every retry costs proxy bandwidth, costs compute, and costs time. And retry rates aren't stable. They spike exactly when the target hardens its defenses, which is exactly when you can least afford it. A pipeline I'd call healthy keeps the retry multiplier low and, more importantly, visible, so you see it climbing before it becomes the whole bill. Most teams don't measure it at all, so the first sign of trouble is the cloud invoice.

Compute is not free just because it's yours

Parsing, rendering, browser automation, the queue workers, the storage of raw payloads. A headless browser is dramatically more expensive than a plain HTTP fetch, and a lot of crawlers reach for the browser by default when most pages don't need one. That's part of why fallback tiers matter: at scale, the difference between "browser for everything" and "browser only when the cheap path fails" is the difference between a sane bill and an absurd one. When you're processing a billion pages a month, a fraction of a cent per page is a real budget line, and the browser-versus-fetch ratio is one of the biggest levers you have.

Anti-bot vendors are a cost center that grows with success

If you're paying a CAPTCHA-solving service or a managed unblocking API, that's per-request and it scales linearly with volume. The trap is that it scales with failed volume too. The harder your targets push back, the more you pay these vendors, the higher your cost per document climbs, and the relationship between "more effort" and "more cost" gets worse precisely on the sources you care most about. Worth watching as its own line, not buried in "infrastructure."

Human babysitting is the cost nobody puts in the model

This is the big one, and it never shows up in a spreadsheet because it's salary, not invoice.

Crawlers break. Sites change layouts, rotate defenses, add a login wall. Someone has to notice, diagnose, and fix it. If that's a senior engineer spending a few hours a week per source across a dozen sources, that's most of a full-time job, and it's the most expensive job on the team. The actual cost of a flaky source isn't the rework. It's the engineer who could have been building something instead of resurrecting a scraper for the fourth time this quarter.

When I model cost per document for a client, the babysitting line is usually the one that changes the decision. A source that looks cheap on proxies but eats six engineer-hours a week is not cheap. It's the most expensive thing you own.

Data quality cleanup, and the source that breaks for good

Two more that ride along. First, dirty data has a cost: dedup, validation, fixing the records that came back malformed. If 10% of what you acquire needs a human or a second pass to be usable, your real cost per usable document is higher than your cost per fetched one, and that gap is the number that matters. Second, replacement. Sources die. A site relaunches, an API closes, a target wins an injunction against scrapers. The cost of building the next acquisition path for that data is part of the cost of the data, and the more sources you depend on, the more often you're paying it.

So what's the number

Add it up and the model looks roughly like this:

cost per usable doc =
    ( proxy + compute + anti-bot vendor ) × retry multiplier
  + ( engineer hours × loaded rate ) / docs per period
  + cleanup cost per doc
  + amortized rebuild cost when the source breaks
  ────────────────────────────────────────────────────
              usable docs delivered

The exact formula matters less than the habit. Most teams track the first line and ignore the rest, which is why their reported cost and their actual cost are different numbers living in different rooms.

Here's the part that surprises people most: once you model it honestly, the rankings flip. The source you thought was cheap, because the proxies were cheap, turns out to be your most expensive data, because it breaks weekly and a senior engineer keeps it alive. The source you thought was expensive runs untouched for months and is actually your best deal per usable record.

You can't make a single good build-versus-buy decision until you've done this math. That's a big part of what the Audit produces: a real cost-per-document number for each source you depend on, and a short list of where the money is actually going versus where you think it is.