Table 1 recomputation and a fertility atlas of nine tokenizers on seven corpora

https://github.com/stw2/tokenizer-science-tax @ 1855bbfabc832bcb964a9b74b90c4c52a7ce50ca

Reference checked 2026-09-14 13:08 UTC. No code was executed or scientific result verified.

• experiments/E01-fertility-atlas/scripts
• experiments/E01-fertility-atlas/data
• experiments/E01-fertility-atlas/DESIGN.md

Prediction

APT4's fertility tax on English scientific corpora exceeds its tax on the English preamble.

Protocol

01 extracts the preamble texts from the paper; 02 fetches the nine tokenizers at pinned revisions; 03 builds seven corpora from pinned dataset revisions and frozen arXiv API responses; 04 recomputes Table 1 under candidate counting protocols and keeps the best-matching one; 05 measures tokens per word, characters per token and bytes per token per corpus with 95% bootstrap intervals and APT4's fertility tax against the Mistral-derived tokenizer; 06 writes flat headline values; 07 writes the content manifest.

Dataset

Preamble of the Constitution of the Republic of Poland in Polish and in its official English translation; seven corpora of about 120k words each (English arXiv abstracts, LaTeX method sections, Python code, GSM8K problems; Polish PES examination questions, Polish Wikipedia science articles, Polish reviews).

Split

None: every document of each corpus is measured.

Access needs

Gated tokenizer repositories need accepted terms and a token; four corpora are not redistributed and rebuild from pinned dataset revisions.

Configurations

Table 1 counting candidates: text variant (flat, lines, protocol), special tokens on or off, word and character counting rules. Bootstrap over documents, 1000 resamples.

Metric

Fertility tax: APT4 tokens per word divided by Mistral-derived tokens per word, per corpus, with a 95% bootstrap interval; Table 1 cells reproduced exactly.

Seeds

20260703 (bootstrap)

Interpretation rule

Refuted if APT4's fertility tax on the English scientific corpora is at most its tax on the English preamble.

Resources

CPU only, hours; no model weights.

Prior work

Table 1 of arXiv:2604.10799v1 reports fertility on the Constitution preamble only.