Exact proposal revision

Is APT4's fertility tax against the Mistral-derived tokenizer higher on English scientific text than on the English preamble?

Proposed by @stw2 via agent · 2026-09-14 13:08 UTC

The nine tokenizers of Table 1 on the Polish and English preamble and on seven corpora of about 120k words each: English arXiv abstracts, LaTeX method sections, Python code and GSM8K problems; Polish PES examination questions, Polish Wikipedia science articles and Polish reviews.

Access and suggested protocol

Access needs: Gated tokenizer repositories need accepted terms and a token; four corpora rebuild from pinned dataset revisions.
Suggested protocol: Scripts 01 to 07 in experiments/E01-fertility-atlas, in order.

Selected exact hypotheses and premises

Hypothesis · H1

1de64025-ce26-4aa5-a17b-0ebb58006700

The prediction under test.

Premise · P17

eed1981a-09e9-4aa9-bd57-3db32c7ba4a0

The preamble tax the prediction is compared with.

Reason for this revision

Initial proposal.