Blog

One Hundred Primary Sources, One Publisher

· NanoParse team

citations is four keys inside the litmus object that arrives with every parse. It answers one question about a page: does the page show its work. The answer is a count of anchors, one count of the subset that sit on a primary host, a per-1,000-word ratio, and a boolean for a references heading.

This post prints the rules that compute them, isolates each one with 33 fixtures, then runs 16 pages through the live endpoint and compares what the counter saw with what the caller actually receives. The comparison is the interesting part: 1,301 anchors were counted across those pages, and 570 link targets appear in the markdown that comes back.

The headline result sits on one page. On nasa.gov, the field reports 100 primary-domain links. All 100 of them point at nasa.gov or one of its subdomains. One hundred primary sources, one publisher.

The field as served

Two response bodies from the run, both from 2026-11-16, copied out of the payloads:

nasa.gov
{"has_references": false, "external_links": 107,
 "external_per_1k_words": 247.1, "primary_domain_links": 100}

docs.python.org/3/library/json.html
{"has_references": false, "external_links": 22,
 "external_per_1k_words": 5.7, "primary_domain_links": 0}

A 435-word front page built from promotional cards scores 247.1 links per 1,000 words and 100 primary links. A 3,852-word language reference scores 5.7 and zero. Neither number is a verdict. The per-1K ratio is a length-normalized count, and the primary count is a hostname test. What follows is the arithmetic behind both.

The function, printed from the deployed source

Two functions, 41 lines, sliced out of worker/worker/src/intel.ts by declaration and run under node --experimental-strip-types for every fixture and recomputation below. The full slice is committed beside the post; what follows is the part that decides all four values.

function isPrimaryDomain(domain: string): boolean {
  if (/\.(gov|edu|mil|int)$/.test(domain)) return true;
  return /(?:journal|research|science|nature|springer|pubmed|plos|nejm|lancet|bmj|ncbi)\./.test(domain);
}

// computeCitations(html, markdown, sourceDomain) — the counting loop
const hrefRe = /href=["'](?:https?:)?\/\/([^\/"'?\s#]+)/gi;
let externalLinks = 0;
let primaryLinks = 0;
let m: RegExpExecArray | null;
while ((m = hrefRe.exec(src)) !== null) {
  const d = m[1].toLowerCase().replace(/^www\./, "");
  if (d === domain) continue; // self-link
  if (/\.(?:png|jpg|jpeg|gif|svg|css|js|woff2?)(?:$|\?)/.test(d)) continue; // assets
  externalLinks++;
  if (isPrimaryDomain(d)) primaryLinks++;
}

const hasReferences = /^#{1,6}\s*(references|bibliography|sources|citations|further reading|works cited)\s*$/im.test(md);

src is the HTML, md is the markdown and domain is the requested host minus www.. The ratio is the accepted count multiplied by ten thousand, divided by the markdown word count and rounded to one decimal place: one accepted link on a 100-word page prints as 10.0, and two on a 300-word page prints as 6.7.

Five properties are visible before any test runs.

The scan reads the HTML, never the markdown. external_links is a count of anchors in the document that was rendered, so links inside navigation, footers, sidebars and card grids are counted. Whether those anchors survive into the markdown the caller receives is a separate question, answered by a separate stage.

The self-link test is exact host equality after dropping one leading www.. nasa.gov skips nasa.gov and www.nasa.gov. It does not skip science.nasa.gov.

The asset filter tests the hostname, not the path. A host called assets.js is skipped. A PNG at upload.example.org/photo.png is counted, because the test never looks at the path.

Primary is a hostname test with two halves: a TLD ending in gov, edu, mil or int, or a substring match for a word from a fixed list immediately followed by a dot. nature.com matches on nature.. springernature.com matches on springer.. Any host with one of those sequences in it is primary, including a publisher's own identity provider.

The heading test runs on the markdown and requires an ATX heading whose entire text is one of six words.

Thirty-three fixtures, one rule each

Each fixture is a minimal HTML string and a markdown string built to test exactly one branch. The columns are the deployed function's output: accepted links, primary links, the ratio, and whether a references heading was found.

fixture                       links  prim   /1k  refs
none                              0     0    0.0  -
one_external                      1     0   10.0  -
self_link                         0     0    0.0  -
self_link_www                     0     0    0.0  -
subdomain_counts                  1     0   10.0  -
protocol_relative                 1     0   10.0  -
root_relative                     0     0    0.0  -
fragment                          0     0    0.0  -
duplicate_hrefs                   3     0   30.0  -
uppercase_href                    1     0   10.0  -
single_quotes                     1     0   10.0  -
port_in_href                      1     0   10.0  -
md_links_not_counted              0     0    0.0  -
gov_primary                       1     1   10.0  -
edu_primary                       1     1   10.0  -
mil_primary                       1     1   10.0  -
int_primary                       1     1   10.0  -
gov_subdomain_primary             1     1   10.0  -
researchgate_not_primary          1     0   10.0  -
sciencedirect_not_primary         1     0   10.0  -
science_word_in_host              1     1   10.0  -
nature_primary                    1     1   10.0  -
nature_word_in_host               1     1   10.0  -
plos_subdomain_primary            1     1   10.0  -
journals_plural_misses            1     0   10.0  -
image_path_counts                 1     0   10.0  -
host_named_asset_skipped          0     0    0.0  -
refs_h2                           0     0    0.0  refs
refs_h3_caps                      0     0    0.0  refs
refs_sources                      0     0    0.0  refs
refs_heading_extra_words          0     0    0.0  -
refs_no_heading                   0     0    0.0  -
per1k_rounding                    2     0    6.7  -

Thirty-three of thirty-three matched the expected output. Six rows are worth reading twice, because each one contradicts a guess a reader would make from the field's name.

researchgate_not_primary links to www.researchgate.net and returns zero primary. The word-list entry is research. with a dot, and researchgate.net has researchg. The same holds for sciencedirect.com. A research platform is not primary by this test, and a hostname that merely begins with a listed word is not either.

nature_word_in_host links to nature.example — no publisher, no journal, a reserved TLD — and returns primary, because the sequence nature. appears in the hostname. science_word_in_host does the same for science.example.org.

journals_plural_misses links to journals.example.org and returns zero primary, while plos_subdomain_primary links to journals.plos.org and returns primary through plos.. The list entry is singular and needs its dot, so the plural fails on the word test and passes only if another entry happens to match.

image_path_counts links to a PNG on a separate host and is counted as an external link. host_named_asset_skipped links to a host called assets.js and is dropped. The filter is a hostname filter that reads like an asset filter.

md_links_not_counted supplies a markdown body containing a markdown link and an HTML body containing no anchors. The count is zero. Only the HTML is scanned.

Sixteen pages, measured

Twenty URLs went through POST /admin/parse on 2026-11-16. Sixteen completed. Four were refused with a named reason and no charge: two content_insufficient (0 words on one, 14 on the other), one consent wall, one bot check. Admin parses bypass quota and payment, so the run cost nothing and touched no free bucket.

The completed set, sorted by length. links and prim are the served payload values; md_ext is the number of link targets in the delivered markdown that point at a host other than the page's own.

page                                  words  links   /1k prim refs md_ext
rfc-editor.org/rfc/rfc9110.html       66479     35   0.5    0    -     32
plato.stanford.edu/…-intelligence     29266     64   2.2   22  yes     60
en.wikipedia.org/wiki/Citation         6848    305  44.6   57  yes    225
en.wikipedia.org/wiki/Web_scraping     4157    109  26.3    1  yes     58
docs.python.org/3/library/json.html    3852     22   5.7    0    -     14
stripe.com/pricing                     3792    249  65.7    0    -     63
vercel.com/pricing                     2166     29  13.4    0    -      0
science.nasa.gov/mars                  1838    158  86.2  148    -     17
kubernetes.io/docs/concepts/overview   1510      5   3.3    0    -      6
nature.com/articles/d41586-024-0…     1385     45  32.8   25    -      8
news.ycombinator.com                    934     33  35.4    2    -     33
arxiv.org/abs/1706.03762                698     46  66.6    0    -     30
docs.anthropic.com/en/docs/overview     633     72 115.2    0    -      3
developer.mozilla.org/…/Fetch_API       470     21  45.5    0    -      4
nasa.gov                                435    107 247.1  100    -     16
example.com                             135      1   7.6    0    -      1
TOTAL                                124598   1301         355    3    570

Read the ratio column top and bottom. The longest document in the run, RFC 9110 at 66,479 words, scores 0.5 links per 1,000 words with 35 external links and no primary hosts. The 435-word NASA home page scores 247.1 with 107 links and 100 primary. On a ratio ranking, the specification reads as unsourced and the front page reads as a research library.

One hundred primary links, four hosts

The primary count on nasa.gov decomposes into four hostnames, all of them NASA's:

science.nasa.gov    88
plus.nasa.gov        7
ciencia.nasa.gov     4
oig.nasa.gov         1
                   ---
                   100

The other seven accepted links on that page sit on seven different hosts, one link each: four social accounts (facebook.com, instagram.com, x.com, youtube.com), an analytics CDN (cdn.parsely.com), a mailing-list form (lp.constantcontactpages.com), and a markup vocabulary site (gmpg.org). So of 107 accepted links, 100 point at NASA's own hosts and the remaining seven point at the agency's social accounts and the tooling that renders its page. Nothing on the page points at another publisher, and the field reports 100 primary-domain links.

science.nasa.gov/mars/ shows the same shape with a different denominator. It returns 158 accepted links and 148 primary, and its six primary hosts are nasa.gov (135), plus.nasa.gov (5), ciencia.nasa.gov (4), eyes.nasa.gov (2), assets.science.nasa.gov (1) and oig.nasa.gov (1). In the page's own source, 187 of its 345 anchors point at science.nasa.gov itself and are skipped as self-links, which is why 158 is well below the markup's anchor count.

A commercial publisher gives the same result from the other direction. The Nature article returns 25 primary links across 14 hosts: 24 of them on springernature.com or a nature.com subdomain — partnerships, authorservices, journal-alerts, cmp, my-profile, idp, mts-nature, press, support, masterclasses, conferences — and one on ncbi.nlm.nih.gov. Two dozen infrastructure links to the publisher's own account pages, counted as primary sources.

The Wikipedia citation article is the case where the field earns its name and still misses the target. It returns 57 primary links across 21 hosts: ncbi.nlm.nih.gov 18, pubmed.ncbi.nlm.nih.gov 15, nature.com 3, lawjournal.mcgill.ca 2 (matched on journal.), and a run of .edu library pages. The two largest destinations on that page are doi.org with 51 anchors and web.archive.org with 34. The DOI resolver — the pointer every citation on the page goes through — is not primary, because doi.org is neither on the TLD list nor contains a listed word.

What reaches the caller

The count is computed on the rendered HTML. The markdown is what survives stripping. The gap between the two columns is the part of the count a reader will never see:

page                                counted  in_markdown  absent
nasa.gov                                107           16      91
science.nasa.gov/mars                   158           17     141
stripe.com/pricing                      249           63     186
vercel.com/pricing                       29            0      29
docs.anthropic.com/en/docs/overview      72            3      69
en.wikipedia.org/wiki/Web_scraping      109           58      51
en.wikipedia.org/wiki/Citation          305          225      80
nature.com/articles/d41586-024-0…        45            8      37
rfc-editor.org/rfc/rfc9110.html          35           32       3
news.ycombinator.com                     33           33       0
TOTAL                                  1301          570     731

Fifty-six percent of the counted anchors are not in the text. On vercel.com/pricing the figure is total: 29 links accepted by the counter, zero external link targets in the markdown. On docs.anthropic.com/en/docs/overview, 72 accepted and 3 retained. On the two Wikipedia articles and RFC 9110 — pages whose links are in the prose — the two columns nearly agree.

The columns come from two reads of the same page. The count is the endpoint's browser render; the markdown is produced from that same render, so it is the honest thing to compare against, but it is not the page source. One row shows the difference in the other direction: kubernetes.io/docs/concepts/overview retains 6 external targets in its markdown against 5 accepted links, because the delivered markdown carries a link the counter's scan of the same render did not accept.

has_references is a heading test

Three of the sixteen pages returned true. The three are the pages whose markdown contains a heading that is exactly References, Bibliography or Further reading:

en.wikipedia.org/wiki/Citation       ## References, ## Further reading
en.wikipedia.org/wiki/Web_scraping   ## References
plato.stanford.edu/…-intelligence    ## Bibliography

RFC 9110 is the counter-example that makes the rule legible. It is a specification with a normative references section, 47 occurrences of the word "references" in its markdown, 35 external links, and three headings in total: RFC 9110: HTTP Semantics, RFC 9110, HTTP Semantics. Its section headings did not survive the conversion as ATX headings, so the test found nothing to match and returned false.

## References and notes returns false, which a fixture covers. The pattern allows the six words and nothing else after them, on the heading line.

Where the render and the source disagree

Every completed page was fetched again with a plain HTTP client and the same two functions were run over that source, with the served markdown as the denominator. external_links and primary_domain_links reproduced exactly on 15 of the 16 pages.

The exception is example.com. The payload reports 1 accepted link; the raw source contains no anchor at all. The delivered markdown carries one link, to iana.org, so the render the endpoint measured had an anchor the fetched source does not. That is the shape of the difference between a browser render and a byte fetch, and it is worth knowing before treating either count as the count.

Using the field without a model call

The four values are enough to route, and not enough to conclude. A gate that reads only the payload:

import re
from urllib.parse import urlsplit

def sourcing(job):
    lit = job["litmus"]["citations"]
    md = job["markdown"]
    hosts = {}
    for t in re.findall(r"\]\((https?://[^)\s]+)\)", md):
        h = (urlsplit(t).hostname or "").lower().removeprefix("www.")
        if h:
            hosts[h] = hosts.get(h, 0) + 1
    page = (job["metadata"]["domain"] or "").lower().removeprefix("www.")
    outside = {h: n for h, n in hosts.items() if h.split(".")[-2:] != page.split(".")[-2:]}
    return {
        "reported_links": lit["external_links"],
        "primary": lit["primary_domain_links"],
        "refs_heading": lit["has_references"],
        "links_in_text": sum(hosts.values()),
        "outside_hosts": len(outside),
        "outside_links": sum(outside.values()),
    }

Run over the sixteen payloads, it produces this:

page              reported prim  in_text  outside_hosts  outside_links
wikipedia.org          305   57      563             55            225
stripe.com             249    0       70              4              5
nasa.gov               158  148       48              3              4
wikipedia.org          109    1      237             37             58
nasa.gov               107  100       37              1              1
anthropic.com           72    0        3              3              3
stanford.edu            64   22       63             45             55
arxiv.org               46    0       44             18             26
nature.com              45   25       31              7              8
rfc-editor.org          35    0      226             11             32
ycombinator.com         33    2       33             29             31
vercel.com              29    0       21              0              0
python.org              22    0       14              7             14
mozilla.org             21    0        4              2              4
kubernetes.io            5    0        6              3              6
example.com              1    0        1              1              1

The NASA home page row is the argument for the pair. The payload reports 100 primary links and 107 accepted links; the text carries 37 links, one host outside nasa.gov, and one link to that host. The Nature article reports 25 primary and carries 8 links outside the publisher's registrable domain. Vercel's pricing page reports 29 accepted links and carries none outside its own.

That is not a defect in the counter, which is doing what its rules say. It is the reason to read two numbers instead of one: the accepted-link count describes the page's markup, and the outside-host count describes the reading a caller actually receives. When they disagree by an order of magnitude, the page is a template with a link quota, not a citation-heavy document.

Reproduce it

Every number above comes from a committed payload, a committed page fetch, or the sliced function. The scripts, in order:

python3 build_fns.py                            # slice the two functions, record blob + sha256
node --experimental-strip-types fixtures.mjs    # 33 fixtures -> results.json
python3 probe_real.py                           # 20 URLs through /admin/parse -> real/
python3 truth.py                                # plain-HTTP fetch of all 16 completed pages
node --experimental-strip-types recompute.mjs   # recompute from each page's own source
python3 rollup.py                               # every table printed above
python3 usage.py                                # the gate snippet, over all 16 payloads

The slice is 41 lines of intel.ts at blob 7b3eca00, source sha256 d773ba97…. Both functions are unchanged since the live deploy. The 33 fixtures, all 16 payloads, the four refusals with their reasons, the 16 page sources, and the rollup output are committed beside this post.

Two scope notes. The real-page run used the admin path, so it cost nothing and consumed no free tier. And every number here describes the deployed function on 2026-11-16; a later version can change any rule above without changing this page.

The endpoint is at https://nanoparse.app — $0.002 per successful parse, no account and no API key, with the same envelope served over MCP at https://nanoparse.app/mcp. Refused reads are never charged. The earlier survey of the six field groups is Was This the Page You Asked For?, and the neighbouring deep dives read one field each: authorship, page_status and structured_data.

← All posts NanoParse home