--- title: "200 Pages, Unedited: What a Parse Endpoint Actually Reaches — NanoParse Blog" url: https://nanoparse.app/blog/2026-09-30-reach-benchmark description: "Every scraping product markets a success rate. Almost none publishes one with the failures attached. So here is a run with all of them: two hundred URLs, parsed…" --- Every scraping product markets a success rate. Almost none publishes one with the failures attached. So here is a run with all of them: two hundred URLs, parsed through the live endpoint on 2026-09-30, with every failure listed and its exact error code. The headline is 40% of the URLs that resolve. That is not a good number, and it is the real one. The more useful finding is that the failures are not random: 79% of them are a site deciding not to serve content to a datacenter IP, and the classes where a reader would most want an answer — news, retail, forums — are the worst of all. ## How the sample was built Two hundred URLs, two samples, both chosen before the run rather than after it. Sample A — the popular web, 100 URLs. The [Tranco](https://tranco-list.eu/) research ranking of the most-visited domains, top 1000, every tenth domain: `rank 1, 11, 21, …`. No curation, no swapping a hard domain for an easy one. Eight domains were excluded on principle (adult, gambling, torrent) before the run; the list ID is recorded with the results. Sample B — a workload, 100 URLs. Six classes an agent is actually asked for: documentation (17), news and blogs (17), government and reference (17), commerce (17), AI-tool and SaaS pricing (16), forums and social (16). Homepages and landing pages — the URL a person pastes into a chat window. One disclosure that matters. Twenty-three of Sample A's domains never resolved: `trafficmanager.net`, `gvt1.com`, `capcutapi.com`, and similar infrastructure, CDN, and telemetry hosts that rank in the top 1000 by traffic and do not serve a page to anybody. Those are counted separately rather than as scraping failures, because they are not. Every percentage below is computed on the 177 URLs that resolved, with the 200-URL denominator printed alongside it so the arithmetic can be checked. ## The numbers ``` nominal resolved usable (>=300 words) all 71/200 36% 71/177 40% 59/177 33% popular homepages (Tranco) 30/100 30% 30/77 39% 21/77 27% workload URLs 41/100 41% 41/100 41% 38/100 38% ``` The two samples land within two points of each other, which is the point: this is a property of the page class, not of the list. A first fifty-URL pass over the same six classes returned 22/50 = 44% — the same ballpark on a smaller sample. By class, with the spread that actually matters: ``` documentation 15/17 88% AI tool / SaaS pricing 8/16 50% government / reference 8/17 47% popular homepages (Tranco) 30/77 39% forums / social 5/16 31% news / blogs 3/17 18% commerce 2/17 12% ``` A parse of a documentation page works almost always. A parse of a retailer or a news site usually does not. Any single success rate quoted for "scraping" is those two numbers averaged together. ## All 106 failures, in five buckets 44 came back `content_insufficient` (42% of failures). The render succeeded and the page's content was not in it — a consent wall, a JS shell, or a stub. Word counts ran from 0 to 72: ``` content_insufficient openai.com/chatgpt/pricing/ "31 words extracted, and the page is a consent wall or a truncated stub (truncated_stub) — this is not the article. You were NOT charged." content_insufficient reuters.com/technology/ "Only 0 words extracted — the page looks like a bot wall, JS shell, or login/security checkpoint. You were NOT charged." ``` 39 came back `blocked` (37%). The site answered with a challenge or a consent interstitial in place of the page: ``` blocked stackoverflow.com/questions "This page is protected by a bot check (cloudflare challenge). You were NOT charged." blocked bankofamerica.com "This page serves a cookie/consent wall instead of its content. We never click through consent banners, so there was nothing real to return. You were NOT charged." ``` 12 render failures (11%) belong to the endpoint, not the site: five navigation timeouts at 8,000 ms, five connection resets, one frame detachment, and one browser session that was still busy. These are retryable, and they are the honest engineering item in this run. 8 policy refusals (8%). The request never left the building: seven paywalled domains and one domain on the known JS-shell/login-wall list. ``` HTTP 403 wsj.com/tech {"error":"Paywalled site — blocked"} ``` 3 not-found or unsupported (3%) — a URL that 404s, and one content type the pipeline will not render. None of the 106 settled a payment. Seventy-one parses were charged at $0.005 each; every failure cost nothing, which is why publishing a run this size is cheap to do. ## The failure that matters is the quiet one The two largest buckets are the same event at different depths. A datacenter IP asks for `chatgpt.com/pricing` and receives the consent interstitial instead of the pricing table. The HTML parses cleanly, the Markdown is real, and it is thirty-one words of cookie banner. That is the failure mode worth being loud about, because it poisons an agent's context quietly: thirty-one words of "we value your privacy" handed to a model that was asked what ChatGPT costs, with nothing in the payload saying the page never arrived. The guard exists to make that loud. A render whose extracted body looks like a consent wall, a JS shell, or a stub is refused, the specific cause is named in `error_code` and `message`, and nothing is billed. A caller gets an error it can branch on rather than prose it has to distrust. ## Reach is not usefulness The 40% above counts a page that came back. It does not count a page worth reading, and the gap is wide enough to state plainly: on the same 177 URLs, the number of parses that returned at least 300 words is 59 — 33%, not 40%. Seven successes came back with 200 words or fewer, and they are the honest boundary cases: - `docs.docker.com/get-started/` — 110 words. A real page, client-side rendered, whose body was mostly chrome. Counted a success. - `nike.com` — 169 words, and 157 words for the same page on `www.nike.com`. Counted a success. Median success was 637 words and the largest was `numpy.org`'s beginner guide at 7,850, so capacity is not the problem. Presence is. Read the Litmus block before trusting a body's length: `content_density` flags thin pages, `paywall` reports truncation, and `reading_time` tells a caller whether there is an article there at all. An endpoint that returns only Markdown leaves that discovery to the caller, at the caller's expense. ## What moves the number, and what is not done Residential proxies are the lever. Route the same requests through home-internet IPs and the bot checks stop firing; published builds using that approach report 90%+ against a datacenter baseline in the thirties. NanoParse does not do that, and the reason is not technical. Reading a page from a residential IP means presenting as a human visitor on a real household connection, and the same vendors sell that traffic for CAPTCHA solving and logged-in scraping. Three lines are held instead: - Consent banners are not clicked. A wall is reported as a wall. - No CAPTCHAs are solved, and no logins are used. - `robots.txt` and Terms of Service are the caller's responsibility; the Terms say so, and a blocklist covers domains whose answer is already known. That is a real limit. This endpoint cannot read most commerce pages, most news, or most of the popular web as ranked by traffic, and this post names them. If the requirement is "read those pages anyway", that is a different product with a different set of risks — and the reference numbers for it live in the marketing of the proxy vendors. ## What a client should do with an error code The five buckets need four different responses, so branch on the code rather than on whether the body is empty: ``` // One parse, branched on error_code instead of on emptiness. const res = await fetch("https://nanoparse.app/fetch", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ url }), }); // 402 → free tier spent or wallet dry; the terms are in the body. // 400 → URL or payload malformed; retrying it unchanged cannot succeed. const { error_code, statusUrl } = await res.json(); switch (error_code) { case "blocked": case "content_insufficient": case "refused_upfront": // The site served a wall, or the domain is a known wall. Terminal for this URL: // fall back to another source rather than retrying. break; case "render_failed": case "timeout": // The failure was on this side of the wire — 12 of 200 in the run above. Retry once. break; } ``` Two rules fall out of the run. A wall is terminal for that URL at that moment — retrying is not a strategy. And an empty or tiny body is not the same as a failure, because the parse that returns 31 words of consent banner reports success on any counter that only checks for non-empty output. ## Re-run it The full result set is committed beside this post: all 200 URLs, both samples, per-URL word counts, error codes, Litmus categories, and the Tranco list ID, plus the scripts that built the sample and ran the parses. Nothing is hand-tuned and the failures are in the file. The run went through an admin path, so it billed nothing; the same script against the public endpoint costs 71 × $0.005 = $0.355 for the same data. An endpoint that publishes its failures is only useful if the number is real, so it is given as measured: 40% of resolving URLs, 33% returning something worth reading, 88% on documentation, 12% on commerce. The next run will be published the same way. ## Getting started Point an MCP client at the hosted endpoint: ``` { "mcpServers": { "nanoparse": { "url": "https://nanoparse.app/mcp" } } } ``` REST clients POST a URL to `https://nanoparse.app/fetch` and follow the returned `statusUrl`. Ten parses per network are free for life, no account and no API key; after that it is $0.005 USDC on Base per successful parse, and a parse that fails, blocks, or lands on a wall is never charged. Docs and the live demo: [nanoparse.app](https://nanoparse.app/) · [quickstart](https://nanoparse.app/quickstart) · [MCP server repo](https://github.com/nanoparse-dev/nanoparse-mcp)