` tag, `md` is the converted markdown, and the probe is those joined by a space. Order matters and the function stops at the first hit, so `kind` is the first wall class a page matched, not the only one it contains. A fixture that says both "sign in to continue" and "verify you are human" returns `captcha`, because that test runs first — correct for a reader, and worth knowing when two walls are present. ## The default is permissive, and confidence describes the branch The last return is the one to read twice. `kind: null` with `confidence: "high"` is what comes back when none of the four patterns matched. It is the absence of a match, formatted to look like a result. `confidence` has two values and it does not mean certainty. The function has five exits: three return `high`, and the `js_shell` and `soft_404` exits return `low`. The label records which branch ran. The page that matched nothing returns `high` because it took the default path — the same label a captcha gets. Reading `confidence: "high"` as "this is confidently a document" inverts the code: on that path nothing was tested and nothing was found. ## The 80-word floor `soft_404` is the only branch with a length condition, and it is `wordCount < 80`. Two fixtures isolate it, identical but for length: ``` fixture words kind conf error soft404_78_words 78 soft_404 low true soft404_85_words 85 null high false ``` The first carries "Page not found" and 78 words and is caught. The second carries the same sentence and 85 words and is not. Seven words separate a detected 404 from a confident document. That floor sits above an earlier one. The refusal layer declines a render under fifty words (`MIN_CHARGEABLE_WORDS = 50`), so a page empty enough to be obvious never reaches this function — it fails as `content_insufficient`. Between 50 and 80 words the soft-404 test is available; above 80 it is not. A 404 page that pads itself with navigation, related links or a search box clears 80 without carrying anything that makes it a page. That is what the Stack Overflow URL does. Its 162 words are almost entirely the similar-questions list. The title matched `page not found` on the first alternative the regex tries, so the phrase test succeeded; the length test then failed, and the function fell through to the default. ## Twenty-two fixtures Each fixture is a title plus a markdown body, run through the sliced function. The last three columns are what the deployed code returns. ``` fixture words kind conf error captcha_verify_human 12 captcha high true captcha_just_a_moment 8 captcha high true captcha_checking_browser 7 captcha high true captcha_prove_human 6 captcha high true login_sign_in_to_continue 7 login_wall high true login_subscribe_to_continue 9 login_wall high true login_create_account 5 login_wall high true js_enable_javascript 7 js_shell low true soft404_title_404 8 soft_404 low true soft404_does_not_exist 5 soft_404 low true soft404_78_words 78 soft_404 low true soft404_85_words 85 null high false document_normal 300 null high false empty_body 0 null high false paywall_phrase 6 null high false cookie_wall 13 null high false login_form_in_html_only 6 null high false signin_nav_with_article 300 null high false code_sample_404_short 17 soft_404 low true quoted_captcha_in_article 400 captcha high true js_phrase_in_docs 300 js_shell low true order_captcha_beats_login 9 captcha high true ``` Twenty-two of twenty-two matched the expected output. The four rows that matter most are the ones where the function behaves exactly as written and the label is still wrong for a reader. ## Phrase tests fire on prose about walls The October survey closed its `page_status` section with a line that reads well and is not accurate: "That is the right scope for the question — a page that discusses captchas is not a captcha." `quoted_captcha_in_article` is a 400-word article about bot walls containing one sentence that quotes a challenge page: a challenge page reads "Just a moment..." and nothing else. The function returns `captcha`, `high`, `is_error_page: true`. The document does not merely discuss captchas; it has been classified as one, because the phrase test cannot tell a description from an instance. `js_phrase_in_docs` is the same shape: a 300-word help page containing "JavaScript must be enabled to use the search box" returns `js_shell`. `code_sample_404_short` is a 17-word code sample titled "Handling HTTP errors" that mentions a 404 and returns `soft_404`. Nothing here is a bug in a phrase counter. It is the reason the output is a signal and not a decision. A page that writes about captchas is more likely to contain the words than a random page is, and four regexes have no way to weigh the difference. ## Wall detection reads the title and the markdown, nothing else `login_form_in_html_only` supplies an HTML document containing a real login form — email field, password field, submit button — with a six-word markdown body. The result is `kind: null`, `confidence: "high"`. The function never looks at the form; it reads `<title>`, joins it with the markdown, and tests phrases. A wall whose labels live only in the markup produces no phrase and therefore no match. That trade is deliberate, and the source records the reason: scanning raw HTML false-positived on clean pages whose scripts mention captcha machinery. The cost sits on the other side, and it is visible in a second fixture. `signin_nav_with_article` is a 300-word article with a "Sign in" link in its navigation and returns `null`, because the pattern wants "sign in to continue", "sign in to read" or "sign in to view" — plain "Sign in" is not one of the alternatives. A site that puts its wall behind an ordinary sign-in button is invisible here. ## Two walls with no kind at all `kind` has four values and the web has more than four kinds of wall: ``` paywall_phrase "This article is for subscribers only." -> null, high cookie_wall "We value your privacy. We and our partners..." -> null, high ``` Neither is one of the classes. A subscription paywall is measured by a different group (`paywall.detected`), and a consent wall is stripped during extraction, which is why a cookie banner rarely corrupts the markdown. `page_status` has no vocabulary for either and returns the default. The real run contains a live example of the same gap. `scribd.com` completed at 260 words with `page_status.kind: null` and `paywall.detected: false`. Its markdown is a free-trial funnel — "Start your free trial", "Free for 30 days, then $11.99/month" — and both fields that could have named it reported nothing. A trial wall is neither a login wall by phrase nor a paywall by that detector's rules. ## Where the refusals land Twenty-two URLs went through `POST /admin/parse` on 2026-11-10. Admin parses bypass quota and payment, so the run cost nothing and consumed no free tier. ``` outcome count completed 7 refused at submit (403) 5 refused during render 10 22 ``` The ten render-time refusals are named: three `not_found`, three `content_insufficient`, three `blocked`, one render failure. The five submit-time refusals are two paywalled domains and three on the domain blocklist, which itself seeds entries as "JS shell / login wall" by class. So most walls are named before `page_status` runs, which is the earlier survey's point and still holds. `example.com/definitely-not-a-real-path-2026` failed as `not_found`; a plain `curl` to that URL returns 404, so the refusal is correct and the field never sees the page. `news.ycombinator.com/login` failed as `content_insufficient` — a login page correctly declined for having too little text rather than mislabelled as a document. The seven that completed are the ones that cleared both layers: ``` words kind conf url 3841 null high docs.python.org/3/library/json.html 3788 null high stripe.com/pricing 2066 null high en.wikipedia.org/wiki/HTTP_404 913 null high news.ycombinator.com 462 null high developer.mozilla.org/.../Fetch_API 260 null high scribd.com 162 null high stackoverflow.com/questions/00000000/not-a-real-question ``` Six of the seven are what they look like. The seventh is the 404 page from the top of this post, and it reached `page_status` at all because the render path produced a body: a plain `curl` to the same URL answers `403` rather than `404`, so a status-code check from a terminal does not see the missing page either. It cleared `content_insufficient` because the similar-questions list is 162 words. That is the precise gap the field could have closed, and the 80-word floor is why it did not. A re-probe of the same seven URLs after this post deployed reproduced six of the word counts exactly, including the 162 on the 404 page. The Hacker News row moved from 913 to 908 in the hour between the two runs — it is a front page, and it rotates. Every `kind` and `confidence` value reproduced unchanged. ## How to read it Five rules, ordered by how much they change a decision. 1. Read `kind: null` as "no wall phrase matched". On the completed set above that meant "document" six times and "404 page" once. 2. Read `confidence` as the branch that ran. `high` on `null` is the default path, not certainty. `low` on `soft_404` and `js_shell` is the function saying it guessed from a phrase and a length. 3. Corroborate with the word count and `content_density`. The Stack Overflow miss is visible in the metadata: 162 words, and the markdown is two headings, one sentence and a link list. A page that clears 80 words on navigation alone has a count without content. 4. Do not treat `is_error_page: false` as clearance. It means no pattern matched, and the four patterns cover captcha, three login phrasings, one script warning and short 404 text. Everything else is outside the vocabulary. 5. When the decision is expensive, read the title. The Stack Overflow URL states the answer in its own `<title>` and again in its first heading. That check is one line, and it would have caught the one real failure in this set. ## Reproducing this The function is not retyped. It is sliced out of the deployed `intel.ts` by declaration, and the slice records the source hash: ``` python3 build_fns.py # slice computePageStatus + record sha256 / blob node --experimental-strip-types probe.mjs # 22 fixtures -> results.json python3 probe_real.py # 14 URLs -> real-pages.json python3 probe_real2.py # 8 more -> real-pages-round2.json python3 rollup.py # every count printed above ``` The slice is 32 lines of `intel.ts` at source sha256 `d773ba97…`, blob `7b3eca00…`, unchanged since the deploy. The fixtures, both real-page runs, the rollup and the scripts are committed beside this post. Two scope notes. The real-page run used the admin path, so it cost nothing and touched no free tier. And every number here describes the deployed function as of this post, not a promise about a later one. The endpoint is [nanoparse.app](https://nanoparse.app/), $0.005 per successful parse, no account and no API key, with the same surface served over MCP at `/mcp`. Refused reads are never charged. The earlier survey this post builds on is [Was This the Page You Asked For?](https://nanoparse.app/blog/2026-10-19-page-you-asked-for), and the neighbouring piece in this series reads one field the same way: [The Author Field Is Present. That Is Not the Same as True.](https://nanoparse.app/blog/2026-11-07-authorship-field) Litmus2,250 words Copy link to this post Test the page your agent just read. Clean Markdown and 20 trust signals, $0.005 per parse. [Run a test](https://nanoparse.app/#parse) On this page Share Copy link Keep reading ## More on litmus. [All posts](https://nanoparse.app/blog) [Nov 13, 2026The Page Says Where It Lives. The Fetcher Echoes What You Asked.A parse returns the Markdown and twenty signal groups beside it. One of those groups, structured_data, is five fields with four booleans and a single string. The booleans say whether a page…](https://nanoparse.app/blog/2026-11-13-canonical-and-fetch) [Nov 7, 2026The Author Field Is Present. That Is Not the Same as True.Litmus returns twenty signal groups in the same response body as the markdown. One of them is authorship, and of the twenty it is the field most likely to be read as a fact about a person: "…](https://nanoparse.app/blog/2026-11-07-authorship-field) ![](https://nanoparse.app/brand/nanoparse-logo-official.svg) ## Your agent deserves better data. Clean Markdown and twenty deterministic trust signals, priced per parse. [Run a free test](https://nanoparse.app/#parse)[See the 20 signals](https://nanoparse.app/litmus) 10 free parses a month · no account

--- title: Page Not Found, Twice, and the Field That Called It a Document | NanoParse url: https://nanoparse.app/blog/2026-11-10-page-status-default description: "An earlier survey of the six signal groups added in October reported that page_status fired on none of ten pages, and drew the right conclusion from it: the…" --- An earlier survey of the six signal groups added in October reported that `page_status` fired on none of ten pages, and drew the right conclusion from it: the wall is usually caught upstream by the refusal path, so this field is the last line. That is true and it is not the whole story. A field that never fires is either unnecessary or wrong, and the difference is measurable. This post measures it. The function is printed from the deployed source and run over 22 minimal fixtures built to isolate each branch, then exercised against 22 real URLs through the live endpoint. The result is a field that works as written, is narrower than its name, and returns the same object for a documentation page and for a 404 page that says "Page not found" in its title and again in its first heading. ## The object as served Two response bodies, both from the live endpoint on 2026-11-10. The first is `example.com`: ``` "page_status": { "is_error_page": false, "kind": null, "confidence": "high" } ``` The second is a Stack Overflow question URL that does not exist: ``` "page_status": { "is_error_page": false, "kind": null, "confidence": "high" } ``` The objects are identical. The pages are not. The second page's served markdown begins: ``` # Page not found - Stack Overflow # Page not found We're sorry, we couldn't find the page you requested. Here are some similar questions that might be relevant: ``` Two headings, both stating the page does not exist, followed by links to questions that do. The field read all of it and answered the same way it answers a reference document. ## Four kinds, checked in order The classifier is four regex tests over one string: ``` const probe = title + " " + md; if (/(verify (?:that )?you (?:are|'re) (?:a )?human|prove you are human|press and hold|checking your browser|just a moment)/i.test(probe)) { return { is_error_page: true, kind: "captcha", confidence: "high" }; } if (/(sign in to (?:continue|read|view)|log in to (?:continue|read|view)|login required|please log in|create an account to (?:continue|read)|subscribe to (?:continue|read)|sign up to (?:continue|read))/i.test(probe)) { return { is_error_page: true, kind: "login_wall", confidence: "high" }; } if (/(enable javascript|javascript is required|please enable js|javascript must be enabled)/i.test(probe)) { return { is_error_page: true, kind: "js_shell", confidence: "low" }; } const wordCount = md.split(/\s+/).filter(w => w.length > 0).length; if (/(404|page not found|not found|no longer exists|does not exist)/i.test(title + " " + md) && wordCount < 80) { return { is_error_page: true, kind: "soft_404", confidence: "low" }; } return { is_error_page: false, kind: null, confidence: "high" }; ``` `title` is the `` tag, `md` is the converted markdown, and the probe is those joined by a space. Order matters and the function stops at the first hit, so `kind` is the first wall class a page matched, not the only one it contains. A fixture that says both "sign in to continue" and "verify you are human" returns `captcha`, because that test runs first — correct for a reader, and worth knowing when two walls are present. ## The default is permissive, and confidence describes the branch The last return is the one to read twice. `kind: null` with `confidence: "high"` is what comes back when none of the four patterns matched. It is the absence of a match, formatted to look like a result. `confidence` has two values and it does not mean certainty. The function has five exits: three return `high`, and the `js_shell` and `soft_404` exits return `low`. The label records which branch ran. The page that matched nothing returns `high` because it took the default path — the same label a captcha gets. Reading `confidence: "high"` as "this is confidently a document" inverts the code: on that path nothing was tested and nothing was found. ## The 80-word floor `soft_404` is the only branch with a length condition, and it is `wordCount < 80`. Two fixtures isolate it, identical but for length: ``` fixture words kind conf error soft404_78_words 78 soft_404 low true soft404_85_words 85 null high false ``` The first carries "Page not found" and 78 words and is caught. The second carries the same sentence and 85 words and is not. Seven words separate a detected 404 from a confident document. That floor sits above an earlier one. The refusal layer declines a render under fifty words (`MIN_CHARGEABLE_WORDS = 50`), so a page empty enough to be obvious never reaches this function — it fails as `content_insufficient`. Between 50 and 80 words the soft-404 test is available; above 80 it is not. A 404 page that pads itself with navigation, related links or a search box clears 80 without carrying anything that makes it a page. That is what the Stack Overflow URL does. Its 162 words are almost entirely the similar-questions list. The title matched `page not found` on the first alternative the regex tries, so the phrase test succeeded; the length test then failed, and the function fell through to the default. ## Twenty-two fixtures Each fixture is a title plus a markdown body, run through the sliced function. The last three columns are what the deployed code returns. ``` fixture words kind conf error captcha_verify_human 12 captcha high true captcha_just_a_moment 8 captcha high true captcha_checking_browser 7 captcha high true captcha_prove_human 6 captcha high true login_sign_in_to_continue 7 login_wall high true login_subscribe_to_continue 9 login_wall high true login_create_account 5 login_wall high true js_enable_javascript 7 js_shell low true soft404_title_404 8 soft_404 low true soft404_does_not_exist 5 soft_404 low true soft404_78_words 78 soft_404 low true soft404_85_words 85 null high false document_normal 300 null high false empty_body 0 null high false paywall_phrase 6 null high false cookie_wall 13 null high false login_form_in_html_only 6 null high false signin_nav_with_article 300 null high false code_sample_404_short 17 soft_404 low true quoted_captcha_in_article 400 captcha high true js_phrase_in_docs 300 js_shell low true order_captcha_beats_login 9 captcha high true ``` Twenty-two of twenty-two matched the expected output. The four rows that matter most are the ones where the function behaves exactly as written and the label is still wrong for a reader. ## Phrase tests fire on prose about walls The October survey closed its `page_status` section with a line that reads well and is not accurate: "That is the right scope for the question — a page that discusses captchas is not a captcha." `quoted_captcha_in_article` is a 400-word article about bot walls containing one sentence that quotes a challenge page: a challenge page reads "Just a moment..." and nothing else. The function returns `captcha`, `high`, `is_error_page: true`. The document does not merely discuss captchas; it has been classified as one, because the phrase test cannot tell a description from an instance. `js_phrase_in_docs` is the same shape: a 300-word help page containing "JavaScript must be enabled to use the search box" returns `js_shell`. `code_sample_404_short` is a 17-word code sample titled "Handling HTTP errors" that mentions a 404 and returns `soft_404`. Nothing here is a bug in a phrase counter. It is the reason the output is a signal and not a decision. A page that writes about captchas is more likely to contain the words than a random page is, and four regexes have no way to weigh the difference. ## Wall detection reads the title and the markdown, nothing else `login_form_in_html_only` supplies an HTML document containing a real login form — email field, password field, submit button — with a six-word markdown body. The result is `kind: null`, `confidence: "high"`. The function never looks at the form; it reads `<title>`, joins it with the markdown, and tests phrases. A wall whose labels live only in the markup produces no phrase and therefore no match. That trade is deliberate, and the source records the reason: scanning raw HTML false-positived on clean pages whose scripts mention captcha machinery. The cost sits on the other side, and it is visible in a second fixture. `signin_nav_with_article` is a 300-word article with a "Sign in" link in its navigation and returns `null`, because the pattern wants "sign in to continue", "sign in to read" or "sign in to view" — plain "Sign in" is not one of the alternatives. A site that puts its wall behind an ordinary sign-in button is invisible here. ## Two walls with no kind at all `kind` has four values and the web has more than four kinds of wall: ``` paywall_phrase "This article is for subscribers only." -> null, high cookie_wall "We value your privacy. We and our partners..." -> null, high ``` Neither is one of the classes. A subscription paywall is measured by a different group (`paywall.detected`), and a consent wall is stripped during extraction, which is why a cookie banner rarely corrupts the markdown. `page_status` has no vocabulary for either and returns the default. The real run contains a live example of the same gap. `scribd.com` completed at 260 words with `page_status.kind: null` and `paywall.detected: false`. Its markdown is a free-trial funnel — "Start your free trial", "Free for 30 days, then $11.99/month" — and both fields that could have named it reported nothing. A trial wall is neither a login wall by phrase nor a paywall by that detector's rules. ## Where the refusals land Twenty-two URLs went through `POST /admin/parse` on 2026-11-10. Admin parses bypass quota and payment, so the run cost nothing and consumed no free tier. ``` outcome count completed 7 refused at submit (403) 5 refused during render 10 22 ``` The ten render-time refusals are named: three `not_found`, three `content_insufficient`, three `blocked`, one render failure. The five submit-time refusals are two paywalled domains and three on the domain blocklist, which itself seeds entries as "JS shell / login wall" by class. So most walls are named before `page_status` runs, which is the earlier survey's point and still holds. `example.com/definitely-not-a-real-path-2026` failed as `not_found`; a plain `curl` to that URL returns 404, so the refusal is correct and the field never sees the page. `news.ycombinator.com/login` failed as `content_insufficient` — a login page correctly declined for having too little text rather than mislabelled as a document. The seven that completed are the ones that cleared both layers: ``` words kind conf url 3841 null high docs.python.org/3/library/json.html 3788 null high stripe.com/pricing 2066 null high en.wikipedia.org/wiki/HTTP_404 913 null high news.ycombinator.com 462 null high developer.mozilla.org/.../Fetch_API 260 null high scribd.com 162 null high stackoverflow.com/questions/00000000/not-a-real-question ``` Six of the seven are what they look like. The seventh is the 404 page from the top of this post, and it reached `page_status` at all because the render path produced a body: a plain `curl` to the same URL answers `403` rather than `404`, so a status-code check from a terminal does not see the missing page either. It cleared `content_insufficient` because the similar-questions list is 162 words. That is the precise gap the field could have closed, and the 80-word floor is why it did not. A re-probe of the same seven URLs after this post deployed reproduced six of the word counts exactly, including the 162 on the 404 page. The Hacker News row moved from 913 to 908 in the hour between the two runs — it is a front page, and it rotates. Every `kind` and `confidence` value reproduced unchanged. ## How to read it Five rules, ordered by how much they change a decision. 1. Read `kind: null` as "no wall phrase matched". On the completed set above that meant "document" six times and "404 page" once. 2. Read `confidence` as the branch that ran. `high` on `null` is the default path, not certainty. `low` on `soft_404` and `js_shell` is the function saying it guessed from a phrase and a length. 3. Corroborate with the word count and `content_density`. The Stack Overflow miss is visible in the metadata: 162 words, and the markdown is two headings, one sentence and a link list. A page that clears 80 words on navigation alone has a count without content. 4. Do not treat `is_error_page: false` as clearance. It means no pattern matched, and the four patterns cover captcha, three login phrasings, one script warning and short 404 text. Everything else is outside the vocabulary. 5. When the decision is expensive, read the title. The Stack Overflow URL states the answer in its own `<title>` and again in its first heading. That check is one line, and it would have caught the one real failure in this set. ## Reproducing this The function is not retyped. It is sliced out of the deployed `intel.ts` by declaration, and the slice records the source hash: ``` python3 build_fns.py # slice computePageStatus + record sha256 / blob node --experimental-strip-types probe.mjs # 22 fixtures -> results.json python3 probe_real.py # 14 URLs -> real-pages.json python3 probe_real2.py # 8 more -> real-pages-round2.json python3 rollup.py # every count printed above ``` The slice is 32 lines of `intel.ts` at source sha256 `d773ba97…`, blob `7b3eca00…`, unchanged since the deploy. The fixtures, both real-page runs, the rollup and the scripts are committed beside this post. Two scope notes. The real-page run used the admin path, so it cost nothing and touched no free tier. And every number here describes the deployed function as of this post, not a promise about a later one. The endpoint is [nanoparse.app](https://nanoparse.app/), $0.005 per successful parse, no account and no API key, with the same surface served over MCP at `/mcp`. Refused reads are never charged. The earlier survey this post builds on is [Was This the Page You Asked For?](https://nanoparse.app/blog/2026-10-19-page-you-asked-for), and the neighbouring piece in this series reads one field the same way: [The Author Field Is Present. That Is Not the Same as True.](https://nanoparse.app/blog/2026-11-07-authorship-field) Litmus2,250 words Copy link to this post Test the page your agent just read. Clean Markdown and 20 trust signals, $0.005 per parse. [Run a test](https://nanoparse.app/#parse) On this page Share Copy link Keep reading ## More on litmus. [All posts](https://nanoparse.app/blog) [Nov 13, 2026The Page Says Where It Lives. The Fetcher Echoes What You Asked.A parse returns the Markdown and twenty signal groups beside it. One of those groups, structured_data, is five fields with four booleans and a single string. The booleans say whether a page…](https://nanoparse.app/blog/2026-11-13-canonical-and-fetch) [Nov 7, 2026The Author Field Is Present. That Is Not the Same as True.Litmus returns twenty signal groups in the same response body as the markdown. One of them is authorship, and of the twenty it is the field most likely to be read as a fact about a person: "…](https://nanoparse.app/blog/2026-11-07-authorship-field) ![](https://nanoparse.app/brand/nanoparse-logo-official.svg) ## Your agent deserves better data. Clean Markdown and twenty deterministic trust signals, priced per parse. [Run a free test](https://nanoparse.app/#parse)[See the 20 signals](https://nanoparse.app/litmus) 10 free parses a month · no account · no API key