What boilerplate_ratio Counts, and What It Cannot See

Litmus returns twenty signal groups in the same response body as the markdown. Two of them look like verdicts on the page: boilerplate_ratio and content_density. Both names promise more than the code delivers, and a developer ranking sources by either will rank them wrong.

Fourteen URLs were submitted to the endpoint on 2026-10-31, across six page classes. Eleven completed, three were refused as blocked. Every payload is committed beside this post, and every number below reproduces from those files — except the word counts on pages that rotate their own content, which moved by a few words on a re-run an hour later while every ratio stayed identical.

The result, sorted by boilerplate_ratio:

ratio  density  words  content_type     url
0.67   0.73     1510   documentation    kubernetes.io/docs/concepts/overview/
0.57   0.93      470   tutorial         developer.mozilla.org/.../Fetch_API
0.27   0.71      555   tutorial         usa.gov/
0.15   0.96      533   academic_paper   en.wikipedia.org/wiki/Boilerplate_(text)
0.09   0.86     4156   tutorial         en.wikipedia.org/wiki/Web_scraping
0.05   0.70     3852   academic_paper   docs.python.org/3/library/json.html
0.02   1.00     1311   news_article     bbc.com/news
0.01   0.81     1375   product_page     stripe.com
0.00   0.50      135   general          example.com
0.00   1.00      949   blog_post        news.ycombinator.com
0.00   0.79      554   documentation    docs.anthropic.com/en/docs/overview

Read as a quality ranking, that table says the Kubernetes documentation page is two-thirds junk and the Hacker News front page is perfectly clean. Both readings are wrong, and the reasons are in twenty lines of source.

What the code counts

Three selectors, one denominator:

const navLen = ($("nav").text() || "").length;
const footerLen = ($("footer").text() || "").length;
const cookieLen = ($('[class*="cookie"], [id*="cookie"], [class*="consent"], [id*="consent"]').text() || "").length;
const totalLen = ($("body").text() || "").length || 1;
const boilerLen = navLen + footerLen + cookieLen;
const ratio = Math.round((boilerLen / totalLen) * 100) / 100;
return { ratio: Math.min(1, ratio), confidence: "high" };

Percentage of all page text that sits inside a

, a

, or an element whose class or id contains cookie or consent. The denominator includes the numerator. The result is rounded to two decimals and capped at 1.

confidence does not grade coverage. It records which branch ran: high means the DOM branch above executed, low means the text fallback did — a repeated-line detector used when no parsed document is available. All eleven completed pages returned high. None of them returned a statement about how much of the page those three selectors found.

Recompute it from the HTML

The number is reproducible. Fetching the raw HTML of five of those pages and applying the same three selectors in Python gives this:

page                  nav  footer  cookie  chars(sel/all)   ratio  published
kubernetes.io/docs     4     1       0       24195/36506     0.66   0.67
docs.python.org/json   2     0       0        1562/30561     0.05   0.05
stripe.com             1     1       4        1852/242888    0.01   0.01
example.com            0     0       0           0/338       0.00   0.00
news.ycombinator.com   0     0       0        0/3959         0.00   0.00

One point apart on the Kubernetes page, because the endpoint measures the rendered DOM and this check counted the raw markup; exact on the other four. So the field is what it looks like: a text-length ratio over three CSS selectors, and anyone can recompute it against the source of the page they paid for.

A zero is the absence of chrome in three places, not the absence of chrome

The Hacker News front page scored 0.00. It is a link index: 949 words in the markdown, 63 of 66 non-empty lines carrying a link. Its layout is nested tables with no

and no

anywhere in the document — the check found zero matching elements, so it had zero characters to count.

The BBC News index scored 0.02 with 47 of its 48 markdown lines being links. Both pages are almost entirely navigation, and both report near-zero boilerplate because the chrome is not inside the three elements the measure reads.

That is not a bug to fix by adding selectors. There is no selector set that identifies "navigation" across the web: a sidebar can be a

, a table, a

, or a shadow root. What the selector set does instead is stay small, cheap and deterministic, and describe exactly one thing — how much text sits in the three places most sites agree to put boilerplate.

The practical consequence: boilerplate_ratio: 0 is not evidence of a clean document. It is evidence that three selectors came back empty, which is true of a pristine article and also true of a page made entirely of links.

A high number is the extraction doing its job

The Kubernetes page scored 0.67, and that one is not a false positive. 24,195 of its 36,506 characters of body text sat inside a nav or a footer, and what survived into the markdown was 10,771 characters.

The order of operations matters here, and the source says it plainly:

// N1 (2026-08-19): cheerioDoc must be the PRE-STRIP DOM (nav/footer intact).
// A stripped doc measures boilerplate as ~0 and mislabels confidence "high".
const boilerplate = computeBoilerplateRatio(markdown, cheerioDoc);

The ratio is measured on the page as rendered, before the markdown is produced. It reports how much of the render did not survive, which is why a documentation site with a large sidebar scores high: two-thirds of the text on that URL is navigation, and the markdown you receive starts after it has been removed.

So the field answers "how much template was on this page", and a high value is a description of the input, not a complaint about the output. The only decision it supports is whether the raw render is worth a second look.

The other field fails in the opposite direction

content_density splits the markdown on sentence punctuation and counts a sentence as substantive if it contains a digit, two adjacent capitalised words, or more than 80 characters:

const sentences = markdown
  .replace(/#+\s+.+/g, "")
  .split(/[.!?]+(?:\s+|$)/)
  .filter(s => s.trim().length > 0);

const hasNumber = (s: string) => /\d/.test(s);
const hasProperNoun = (s: string) => /[A-Z][a-z]+\s+[A-Z][a-z]+/.test(s);

for (const sent of sentences) {
  if (hasNumber(sent) || hasProperNoun(sent) || sent.length > 80) substantiveCount++;
}
const ratio = substantiveCount / sentences.length;
const thinContent = ratio < 0.25;

Across the eleven completed pages the ratio ranged from 0.50 to 1.00, and thin_content was false eleven times. Two pages scored exactly 1.00: Hacker News and the BBC index. Both are link lists. Split a block of link labels on [.!?] and there are almost no boundaries to split on, so the whole list arrives as one or two enormous "sentences", each far past 80 characters, each counted as substantive.

The field is honest about what it computes and easy to misread anyway: on prose it measures sentence shape, and on punctuation-free text it measures nothing at all while returning a perfect score.

Volume is not its job either. example.com returned 135 words and scored 0.50 with thin_content: false. A five-paragraph page clears the 0.25 bar because its sentences are long, not because 135 words is enough to cite. The floor on volume lives somewhere else entirely, in the refusal layer, where a render under fifty words fails instead of completing:

// x402-v2.ts
export const MIN_CHARGEABLE_WORDS = 50;

export function isChargeableContent(markdown: string): boolean {
  const words = markdown.split(/\s+/).filter((w: string) => w.length > 0);
  return words.length >= MIN_CHARGEABLE_WORDS;
}

Two different questions, two different layers, and neither one is "is this a good source".

Which field answers which question

The twenty groups are not a scorecard, and reading them as one is how a pipeline ends up preferring a link index over a documentation page. Each field answers a narrow question:

question                              field
was the page mostly template?         boilerplate_ratio.ratio (high = yes)
is this a document at all?            page_status.kind (soft_404, login_wall, captcha, js_shell)
is the text prose or fragments?       content_density.substantive_sentence_ratio
is there enough text to be useful?    the word count in the envelope, and the 50-word floor
does it cite anything?                citations.external_links, has_references
who wrote it?                         authorship.name, kind, source
is it current?                        freshness.published, age, stale_warning

A keep-or-drop check that uses them honestly needs one conjunction and no ranking:

// keep.ts — is this envelope worth storing as a source?
export function keepAsSource(litmus: any, words: number) {
  const kind = litmus.page_status?.kind;
  if (kind) return { keep: false, why: `not a document: ${kind}` };

  if (words < 50) return { keep: false, why: `too little text: ${words} words` };

  if (litmus.content_density?.thin_content) {
    return { keep: false, why: "no substantive sentences" };
  }

  // The only place the ratio earns a vote, and only with a second condition:
  // a template-heavy render that also produced little text is a menu.
  if (litmus.boilerplate_ratio?.ratio > 0.5 && words < 300) {
    return { keep: false, why: "mostly template and little content" };
  }

  return { keep: true, why: "document" };
}

Note what is not in that function: no score, no weighting, no sorting. A page that fails none of the four checks is kept, and the fields that decide whether it is the right page — content_type, citations, authorship, freshness — are read afterwards, by whatever downstream code actually cares.

Descriptive, not editorial

Every field in the envelope describes the render that produced the markdown: selectors that matched, sentences that split, links that counted. None of them ranks the page, and the two that sound most like rankings are the two most easily misread.

The measurement is worth repeating on any signal you plan to filter on. Take ten pages of the class you actually parse, read the field against the payload, and recompute it once by hand. If it reproduces, it is a fact about the input and safe to use as a gate. If it does not, the field is doing something narrower than its name, and a filter built on it will silently drop the pages you wanted.

Price for a parse that completes is $0.005 per page, refused reads are never charged, and the fields above arrive in the same body at no extra cost. The endpoint and its MCP surface are described at nanoparse.app/quickstart, the full signal list at nanoparse.app/litmus, and the machine-readable contract at nanoparse.app/llms.txt. A related read on treating one of these fields as a ledger: Nothing Fired.