Why your scraper works in dev and dies in prod

Your scraper works on your laptop. Of course it does. You built it there, pointed it at a handful of pages you were watching, saw clean rows come out, and shipped it. That isn't proof it works. It's proof it survives the easiest conditions it will ever face.

Then it goes to production and starts lying to you. Not crashing, lying: reporting success while quietly collecting nothing. The code didn't change. The environment did, and for a scraper the environment is the entire game.

I've watched this exact failure for a decade, running acquisition at ZoomInfo and now in audits, and it is almost never a bug. It's a short list of differences between your machine and production that don't matter for most software and matter enormously for a scraper. Here are the ones that actually kill you.

Your laptop has a reputation your servers don't

The single biggest difference between dev and prod is the IP address, and nobody thinks about it until it's too late. Your laptop sits behind a residential ISP connection. To an anti-bot system that IP looks like a person, because it is one. Production runs from a cloud datacenter, and every serious anti-bot vendor keeps lists of AWS, GCP, and Azure ranges and treats them as guilty until proven innocent.

So you can ship byte-for-byte identical code, the same headers, the same request pattern, and get a clean 200 on your laptop and a hard block from the datacenter. The scraper didn't break. It walked up to the same door wearing a uniform the bouncer doesn't like. This is why "it works locally" tells you close to nothing about a defended target: you tested from the one IP on earth least likely to get blocked.

Volume is a different animal, not a bigger one

In dev you hit the target maybe fifty times. In prod you hit it fifty thousand times, and a lot of a site's defenses don't wake up until you cross a threshold you never came close to on your machine. Rate limits, velocity checks, behavioral flags: they are all watching for volume, and a trickle sails under every one of them.

This is the trap that feels the most unfair, because scaling up is supposed to be the boring part. You proved the logic works, now you just run it more. But "run it more" is precisely the thing the target is defending against. The tenth request and the ten-thousandth request are not the same request to the site, even though they're identical to you. More isn't just more. More is what trips the wire.

You tested the pages you wish existed

When you build a parser you point it at a few example pages, and naturally you pick good ones: fully populated, current, well-formed. Production doesn't get to pick. It hits the listing with a missing price, the profile with a field that's blank this week, the "this item was removed" tombstone, the page that quietly got an A/B-tested layout, the record with a unicode character your parser has never met, the country redirect you didn't know existed.

Your parser encoded assumptions about structure that were true for your five examples and false for the long tail. And the long tail is enormous. On a real crawl the weird page is not an edge case you'll get to later. It's ten percent of your volume, and it's the ten percent that decides whether anyone can trust the dataset.

Dev is a moment. Prod is time.

You run a scraper in dev once and watch it finish. Production runs it forever, and forever introduces enemies a single execution never meets. Session cookies expire mid-crawl. Auth tokens rot. Pagination cursors drift. The site ships a layout change at 2am on a Tuesday and your selectors point at nothing by breakfast. None of this shows up in a thirty-second dev run, because none of it has had time to happen yet.

A scraper that works is really a scraper that works right now. Whether it still works next month is a completely different property, and it's the one that matters, because the entire point of production is that it keeps running when you are not looking.

The failure that returns 200

Here is the one that does the real damage, and it's the reason "it works in dev" is so dangerous. In dev, when something breaks, you see it, because you're staring at the output. In prod the nastiest failures don't announce themselves. The target returns a 200 OK with a CAPTCHA page in the body. Or an empty results container. Or a polite "we noticed unusual activity" soft block that looks, to your code, exactly like a normal page.

Your scraper doesn't crash. It parses that page, extracts nothing or garbage, writes it to the database, and reports success. Every dashboard is green. The pipeline is "healthy." And you are now confidently accumulating empty rows, which you'll discover three weeks later when someone senior asks why the numbers look wrong. A scraper that crashes is a good scraper: it's telling you the truth. The one that smiles and returns nothing is the one that quietly costs you a quarter.

In dev, you are the monitoring

Notice the thread running through all of this. In dev, you're watching. You are, personally, the health check, the anomaly detector, and the alerting system. You catch the block, you notice the empty output, you spot the layout change because you happen to have the page open.

Production has none of that unless you build it, and most scrapers ship without it, because monitoring feels like overhead bolted onto a thing that already works. But "it already works" was only ever true while you were watching. The moment you look away, an unmonitored scraper stops being a data source and becomes a rumor. You have to instrument it to tell you when it's blocked, when volume drops, when the shape of the output shifts, because the target will not do you the courtesy of failing loudly.

How to make dev tell the truth

The fix for most of this has one shape: stop trusting the easy environment. A surprising amount of the gap closes the moment you make dev resemble prod before you believe it. Run your pre-ship test from the IP you'll actually deploy on, not your laptop. Push enough requests to cross the thresholds a trickle never reaches. Point the parser at a saved set of the ugliest pages you can find, the removed listings and the blank fields and the strange layouts, instead of the five clean ones you started with. Almost every "but it worked in dev" surprise is a page or a condition you quietly declined to test.

The one thing testing alone won't save you from is silence, so you instrument for it directly. Treat an empty result as a failure, not a success: zero rows should page someone, not pass quietly. Assert on the shape of the output instead of just the HTTP status, required fields present, row counts inside an expected band, a canary page whose correct answer you already know so a wrong one screams. And alert on deltas, because the target won't flag itself for you: a fill rate that craters or a daily volume that halves overnight is the sound your scraper makes when it's being blocked and still returning 200. None of this makes scraping easy. It makes it honest, which is the only version that survives contact with production.

A scraper isn't done when it works

Every failure above comes from the same mistake: treating "it ran on my laptop" as the finish line, when it's barely the starting one. Dev proves your parsing logic is plausible. It proves nothing about IP reputation, volume, the long tail, time, or silence, and those five things are what production is made of.

A scraper is done when it survives datacenter IPs, holds up under real volume, handles the ugly pages, keeps working after the site changes underneath it, and tells you the moment it stops. That's not a bigger script. It's a different kind of thing: a system with fallbacks, monitoring, and the standing assumption that the target is actively trying to stop it. It's the same shift that separates the projects that last from the ones that die in month three. Build the system, and "works in dev" becomes what it always should have been: the least interesting thing you can say about a scraper.